How to quickly setup a load balanced, high availability, Apache cluster

I wanted to find a simple to maintain and expand soultion for load balancing a web cluster with high availability. I have found my solution in HAProxy.

Scenario

Bluhalo Labs HAProxy Test Rig

This demo scenario is in the following enviroment:

• Network Configuration
  • Network: 192.168.11.0
  • Subnet Mask: 255.255.255.0
  • Gateway: 192.168.11.254 (Although this is irrelavant)
• 1 x Shared IP for the Load Balancers
  • 192.168.11.40
• 2 x Load Balancers
  • BHLabs1 – 192.168.11.30
  • BHLabs6 – 192.168.11.39
• 4 x Web Servers
  • BHLabs2 – 192.168.11.35
  • BHLabs3 – 192.168.11.36
  • BHLabs4 – 192.168.11.37
  • BHLabs5 – 192.168.11.38

At the start of this setup all machines are running Ubuntu 8.04 Server from a standard install with openssh-server installed and the root password set. All setup commands are run as root or with sudo.

Web Server Configuration

Basic Config

As we are only doing this as a basic test, a very simple Apache config is required.
This will install Apache 2 and also PHP5 to give us some basic scripting to output server name for testing etc that you may wish to play with later.

# apt-get -y install php5

Next you need to create a check file for HAProxy to look for from the load balancers. This file will be used to determine if the servers are up. This will create a blank file called check.txt in the default DocumentRoot for Apache.

# touch /var/www/check.txt

Now stick your test index.html in that directory as well.

# echo "oh hi" > /var/www/index.html

Log Modification

Filtering out HAProxy health checks

You don’t want to log hits to the check.txt file in your Apache logs, so put an exclusion in your VirtualHost directive. here’s an example of how:

<VirtualHost *>
  ServerAdmin server-alert@bluhalo.com
  DocumentRoot /var/www/
  ErrorLog /var/log/apache2/error.log
  LogLevel warn
  CustomLog /var/log/apache2/access.log combined env=!dontlog
  SetEnvIf Request_URI "^/check\.txt$" dontlog
</VirtualHost>

Modify the access logs

HAProxy will act as a completely transparent proxy so by default the web servers will log the load balancers IP in it’s logs instead of the user’s. HAProxy add’s the user’s IP to the header in the “X-Forwarded-For” field, so you need to modify the log configuration in your apache2.conf to take advantage of this:

vi /etc/apache2/apache2.conf

Search for entries that start “LogFormat” …

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent

… and swap “%h” for “%{X-Forwarded-For}i” like:

LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent

Load Balancer Configuration

You need to install HAProxy and Heartbeat for this setup to work. HAProxy provides your load balancing functionality and Heartbeat provides your high-availability failover functionality.

# apt-get -y install haproxy heartbeat-2

Let’s start with HAProxy as thats the easier one :)

HAProxy

Open up the HAProxy config file:

# vi /etc/haproxy.cfg

and replace the whole file on both servers with the following:

global
  log 127.0.0.1 local0
  log 127.0.0.1 local1 notice
  maxconn 4096
  #debug
  #quiet
  user haproxy
  group haproxy

defaults
  log global
  mode  http
  option  httplog
  option  dontlognull
  retries 3
  redispatch
  maxconn 2000
  contimeout  5000
  clitimeout  50000
  srvtimeout  50000

listen bhlabslb 192.168.11.40:80
  mode http
  stats enable
  stats auth admin:password
  balance roundrobin
  option httpclose
  option forwardfor
  option httpchk HEAD /check.txt HTTP/1.0
  server  inst1 192.168.11.35:80 cookie server01 check inter 2000 fall 3
  server  inst2 192.168.11.36:80 cookie server02 check inter 2000 fall 3
  server  inst3 192.168.11.37:80 cookie server01 check inter 2000 fall 3
  server  inst4 192.168.11.38:80 cookie server02 check inter 2000 fall 3
  capture cookie vgnvisitor= len 32

  rspidel ^Set-cookie:\ IP= # do not let this cookie tell our internal IP address

You also have to allow the HAProxy service to start. Change the ENABLED value in the /etc/default/haproxy:

# Set ENABLED to 1 if you want the init script to start haproxy.
ENABLED=1
# Add extra flags here.
#EXTRAOPTS="-de -m 16"

Heartbeat

We will be using Heartbeat to pass the shared IP address (192.168.11.40) between our 2 load balancers if one goes down. To do this, it needs to be able to bind to an address that doesn’t yet exists on the system. In order to allow this you need to add the following to /etc/sysctl.conf:

# Allow HAProxy shared IP
net.ipv4.ip_nonlocal_bind = 1

and then run:

# sysctl -p

Heartbeat requires 3 main configuration files which do not come with the install. First of all the authkey. Do the following on both servers:

# vi /etc/ha.d/authkeys

add the following content, making sure you replace MyPassword with a secure string. This needs to be the same on both servers:

auth 3
3 md5 MyPassword

This file MUST be accessible only by root or Heartbeat won’t start:

# chmod 600 /etc/ha.d/authkeys

Next on each server create the following. Run:

# uname -n

to get the kernels take on the local hostname, and then insert this into:

# vi /etc/ha.d/haresources

in the following syntax:

BHLabs1 192.168.11.40

and

BHLabs6 192.168.11.40

Note the hostname changes but not the IP. The IP is the shared IP. Finally the main Heartbeat config file:

vi /etc/ha.d/ha.cf

On the first server:

#
#       keepalive: how many seconds between heartbeats
#
keepalive 2
#
#       deadtime: seconds-to-declare-host-dead
#
deadtime 10
#
#       What UDP port to use for udp or ppp-udp communication?
#
udpport        694
bcast  eth0
mcast eth0 225.0.0.1 694 1 0
ucast eth0 192.168.11.30
#       What interfaces to heartbeat over?
udp     eth0
#
#       Facility to use for syslog()/logger (alternative to log/debugfile)
#
logfacility     local0
#
#       Tell what machines are in the cluster
#       node    nodename ...    -- must match uname -n
node    BHLabs1
node    BHLabs6

and on the second server:

#
#       keepalive: how many seconds between heartbeats
#
keepalive 2
#
#       deadtime: seconds-to-declare-host-dead
#
deadtime 10
#
#       What UDP port to use for udp or ppp-udp communication?
#
udpport        694
bcast  eth0
mcast eth0 225.0.0.1 694 1 0
ucast eth0 192.168.11.39
#       What interfaces to heartbeat over?
udp     eth0
#
#       Facility to use for syslog()/logger (alternative to log/debugfile)
#
logfacility     local0
#
#       Tell what machines are in the cluster
#       node    nodename ...    -- must match uname -n
node    BHLabs1
node    BHLabs6

Testing

Restart all the services. On the web servers run:

# apache2ctl restart

and on the load balancers:

# /etc/init.d/heartbeat start
# /etc/init.d/haproxy start

You should then be able to hit http://192.168.11.40 and see your test webpage!
You should also have a page full of stats to please they eyes from HAProxy at http://192.168.11.40:81/haproxy?stats. This can be turned off by removing the following 2 lines from the haproxy.cfg. This should be removed in a production enviroment.:

 stats enable
 stats auth admin:password

Try the following:

• Kill the heartbeat service on each load balancer in turn while running “watch ifconfig”. You should see the IP address move from server to server.
• Pull the plug on either of the load balancers while running a ping from your PC to 192.168.11.40, the ping should never fail.
• Shut down Apache on any of the web servers, you should see them go red in the stats page within 2 seconds and they will be removed from the load balanced group until they come back up.

Caveats and best practices

This setup is lacking in some important best practices:

1. The heartbeat should run on it’s own network, usually a VLAN dedicated to it.
2. The web servers should sit on a different network to the frontend on the load balancers. The load balancers would then have a second interface out to that network to connect to the web servers.

Later I will follow this up with a way to add MySQL to this configuration.

Reference

• HAProxy documentation: http://haproxy.1wt.eu/download/1.3/doc/configuration.txt
• Heartbeat documentation: http://www.linux-ha.org/LearningAboutHeartbeat

---

Possibly related posts: (automatically generated)

• Keep your Web site online with a High Availability Linux Apache cluster
• Load Balance and Cluster Setup for Tomcat via Apache 2.2
• HA Linux Apache cluster