Working in the AWS public cloud; one has to adapt to a world of guaranteed failure at unpredictable times. Utilizing a combination of LTM for HA within a single availability zone and GTM across availability zones and regions provides an architecture to survive the chaos monkeys.

The following is part of a demo environment that I built for AWS, it highlights a couple of useful features of LTM/GTM including:

  • Monitoring LTM services from GTM
  • Using GTM for outbound DNS resolution
  • Creating split DNS records for EIP vs. internal VPC IP
  • Create topology LB records to avoid cross-AZ communication for active-active services

Demo

Here’s what the overall architecture looks like:

clip_image002

We create a couple of wide IP records to show different failure scenarios (these DNS records are isolated to my demo environment):

  • prefer-d.f5demo.com
    • Active/Standby from D to E
  • prefer-e.f5demo.com
    • Active/Standby from E to D
  • active-active.f5demo.com
    • Round robin between D and E (unless internal request)

Here’s what things look like when everything is OK from an external user:

clip_image004

Or another view using a command line view with curl:

clip_image005

Taking a look from an internal user we can see that the behavior is slightly different. In this case a request from the US-EAST-1D AZ will always request from the same AZ when communicating with the active-active service. It also accesses the service using the internal IP address. This provides some cost savings if you have services that are very data heavy to avoid cross-AZ data billing charges.

clip_image006

Performing a failure of the D services (stopping the web server) we can see that initially connections fail while the client is still trying to access services use the US-EAST-1D IP address:

clip_image007

Once the client refreshes its DNS record (default 30 second TTL) we can see that it is now only communicating with US-EAST-E services:

clip_image008

Setting it up

To create this demo you’ll need:

  • LTM/GTM devices
  • Two AZ
  • Some backend services

Once you have these you can build out a standard LTM/GTM environment in AWS. Create a DNS cache (required for Topology LB). Point your AWS instances to your DNS cache listeners (make sure this is only accessible to your internal clients!!!). Build up some Split DNS / topology records.

You can also extend this example to go cross-Region as well (with the limitation that your internal IP space would not be accessible cross VPC)!

More Chaos

The example above illustrates a single failure scenario (loss of US-EAST-1D web services). You can imagine that there’s several other scenarios that could cause a failure including and not limited to:

  • Loss of EAST-1E services
  • Loss of a single AZ
  • Loss of a single LTM/GTM device

The demo environment is built to survive these and keep on running despite the best efforts of any chaos monkeys.