Zen Load Balancer
I am using Zen Load Balancer as the front end to all of the infrastructure in this project. It is simple, robust, provides a nice air gap between application servers and the big bad world and it also doubles as a choke point that we can apply security policies and traffic monitoring when required.
I chose Zen over better known load balancers as it can handle TCP and UDP and it will also do SSL offloading, freeing up some CPU on the app servers if we need it later on.
Zen Load Balancer will sit in front of every service we allow to be seen by the outside world – HTTP, HTTPS, DNS, SMTP, POP, IMAP.
DNS Load Balancing
I am also using Amazon’s Route 53 DNS service. They have a feature that allows me to perform health checks on a URL or TCP port. In the event of that URL or service failing, it withdraws the IP from production. I’m planning to use a single A record with each load balancer’s IP in it. If a load balancer fails the health check, it is pulled from the DNS.
Each domain I am hosting will use www CNAME’s that point to the load balancer DNS record. Unfortunately base domain A records will still need to be setup manually as there are a bunch of reasons (email delivery particularly) that you can’t use a cname for the domain name itself.
I am also going to build my own DNS infrastructure for client & friends domains that I host. That infrastructure will use its own set of tricks to simplify the effort required to setup and manage domains and to minimise the number of domains I’ve got to pay Amazon to host.
Setting up Zen
Setting up Zen Load Balancer is incredibly easy, I followed the config guide here without any issues. Remember that you will need a public IP for eth0 and you’ll want to use a private IP on eth1.
I’ve setup two load balancers per island with a public IP address each. I’ve not used the built-in clustering & virtual IP stuff for the moment as I think the AWS DNS health check system is going to do the job a little better.
Something semi important to be aware of that had me scratching my head for a while: Zen does not use the standard /etc/network/interfaces that Debian and Ubuntu use, it takes control of the network interfaces at boot, so if you’re planning to change stuff (like routes to additional internal IP ranges) you’ll need to do it in the Zen config files and restart Zen from the command line.
Lock down the Web GUI
Like any other admin interface, the first thing you should do when the box first boots is lock it down.
1. Change the admin password.
2. Bind the GUI to the internal interface only.
3. Bind SSH to the internal interface only.
Open /etc/ssh/sshd_config in your favourite text editor and find this line (you will need to su to root first):
Change it to this, where x.x.x.x is the internal ip address of the machine you’re configuring.
Once you’ve done that, restart SSH
service ssh restart
Farms are what Zen calls services. Every service/port you want to be load balanced will require a farm to be setup.
Initial Farm Setup
When you first configure a farm, you’re presented with a series of options, the defaults should be good enough for our needs.
Real Server Setup
Real servers are the backend boxes you’re going to be pointing the load balancer at, what we’re calling the app servers in this project. You will notice that there is an option to put a server into “Maintenance Mode” this is a handy way to take a box out of production gracefully if you need to, you can see I have 10.0.0.20 in Maintenance Mode now.
Something to bear in mind is you can even use servers that are not in your private network, so you could load balance to a machine on a public IP in a totally different location if you choose to, this is something I’m planning to take advantage of when migrating services onto the new platform. I will set the load balancer up to point at the old servers but move DNS records to point at the load balancers, then when I want to swap to the new infrastructure, I can do so at a moments notice and switch back quickly if I need to as well. You could also use this setup to point your site at some new AWS instances you’ve just spun up because your website is being smashed by some unexpected transient traffic (clickfrenzy anyone?).
So you repeat this process for all the services you want to have the load balancers manage. You can use different real servers or groups of real servers for each service if you want, or you can make every backend box do every task. This gives you a huge amount of flexibility and the ability to scale up very quickly if you want to.
This is what my configured farms look like.
Setting up Amazon DNS
Setting up Amazon DNS is really easy, Amazon’s Route 53 console has been built quite cleverly, it even stops you from making mistakes that break RFCs and cause downtime or unpredictable behaviour.
Once you’ve created your domain, you will need to create the load balancer health checks.
To create the health checks, go to the “Health Checks” section of the console on the left, click “Create Health Check”. Leave it as an “HTTP” check, enter the public IP address of your first load balancer, leave the port as 80, enter the hostname of the load balancer and type “test.txt” in as the path.
This will make Amazon check the load balancer every 30 seconds for a file called test.txt, if it can’t get that file, it will assume the load balancer isn’t healthy and remove it from production. The file test.txt will be sitting on a wildcard vhost we will create when building the web servers later on.
Repeat this for all of your HTTP load balancers. You can so the same for SMTP POP3 etc if you choose as well.
Create A records
Creating load balanced & health checked A records is a little different to normal A records in Route 53. Because we want each load balancer IP address to have its own dedicated health check, we have to create multiple A records, something thats not possible in Route 53 unless you’re using a routing policy.
Delete old records
If you’ve already got an A record for the DNS entry you’re about to create, you’ve got to delete it before you try to create the new one or Route 53 spits the dummy. I suggest creating a whole new entry like lb.domain.com to test before going too crazy.
Create new A record
- Click “Create Record Set”
- Leave the Type as “A – IPv4 address”
- Alias = no
- Set the TTL to 60. You want a short TTL on this record for failover to work properly
- Put the public IP of your first load balancer in the Value field
- Change Routing Policy to “Weighted”
- Set weight to “1”
- Enter “Primary Load Balancer” into “Set ID”
- Set “Associate with Health Check” to Yes
- Select the health check you setup earlier for this load balancer from the “Health Check to Associate” pulldown.
- Hit “Save Record Set”
- Repeat this for each load balancer you want to put into production.
To test this you’re really going to need to have a web server serving up test.txt. If you can point your load balancers at such a beast, then you can test the failover by executing “dig a lb.domain.com” from a linux command line. It will randomly return all active load balancers IP addresses. If one goes down you should see that IP address stop being served by Amazon within 90 seconds.
Next week we are building redundant file servers (its been a bit of a nightmare, definitely worth the read!), if you’ve got any questions or comments, please remember to comment below!