Scenario

Background

This post is a follow on from my previous post on how to setup a load balanced high availability Apache cluster and uses the same network setup and heartbeat setup. For the purposes of testing and to save on resource I am using the web servers from the previous guide as the MySQL servers in this guide. Obviously in a real world setup this would be doing it wrong

End Goal

You will be able to connect to 1 IP address and port with MySQL client and be routed to 1 of many MySQL servers. The machine doing the routing will be in a high availability setup with another machine.

Server Roles

bhlabs1/6 will load balance mysql queries through mysql-proxy
bhlabs2 will be  the master replication server
bhlabs3-5 will be mysql slaves

I’m separating this guide into 3 sections:

• MySQL Replication Cluster
• mysql-proxy Setup
• Usage with heartbeat

MySQL Replication Cluster

On each of the servers :

# apt-get update
# apt-get install mysql-server

The slave servers will need special access to stream the binary logs from the master server, so that needs to be setup using the following GRANT command as root on the master server:

mysql> GRANT REPLICATION SLAVE ON *.* TO 'repl'@'%' IDENTIFIED BY 'ImASlave';

Next un-comment server id and binary log entries in my.cnf (add them if they weren’t there in the first place):

server-id   = 1
log_bin     = /var/log/mysql/mysql-bin.log

You must also set the server-id on all slaves, starting from 2 and incrementing as you go. You do not need the log_bin entry on the slave servers.

Your slave servers must begin replication at exactly the same state as the master server. From the master server, export all databases with lock on all tables:

# mysqldump -p --all-databases --lock-all-tables > initialDbDump.db

And then import on to all slave servers:

# mysql -p -hbhlabs3 < ./initialDbDump.db
# mysql -p -hbhlabs4 < ./initialDbDump.db
# mysql -p -hbhlabs5 < ./initialDbDump.db

Get log position off bhlabs2:

mysql> show master status;
+------------------+----------+--------------+------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+------------------+----------+--------------+------------------+
| mysql-bin.000002 |      516 |              |                  |
+------------------+----------+--------------+------------------+
1 row in set (0.00 sec)

Set configuration and start slave on bhlabs3-5:

mysql> CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.000002', MASTER_LOG_POS=516, MASTER_HOST='bhlabs2', MASTER_USER='repl', MASTER_PASSWORD='ImASlave';
mysql> START SLAVE;

You can check connectivity from bhlabs2:

mysql> show full processlist;
+----+------+---------------------+------+-------------+------+----------------------------------------------------------------+-----------------------+
| Id | User | Host                | db   | Command     | Time | State                                                          | Info                  |
+----+------+---------------------+------+-------------+------+----------------------------------------------------------------+-----------------------+
| 16 | repl | 192.168.11.36:40812 | NULL | Binlog Dump |  248 | Has sent all binlog to slave; waiting for binlog to be updated | NULL                  |
| 17 | repl | 192.168.11.37:48080 | NULL | Binlog Dump |  206 | Has sent all binlog to slave; waiting for binlog to be updated | NULL                  |
| 18 | repl | 192.168.11.38:54400 | NULL | Binlog Dump |  202 | Has sent all binlog to slave; waiting for binlog to be updated | NULL                  |
| 22 | root | localhost           | NULL | Query       |    0 | NULL                                                           | show full processlist |
+----+------+---------------------+------+-------------+------+----------------------------------------------------------------+-----------------------+
4 rows in set (0.00 sec)

And then create a test database on bhlabs2 and it should appear on the other 3 servers.

mysql> create database test;
Query OK, 1 row affected (0.00 sec)

Drop the test database and it should disappear. If this works you’re set to go.

mysql> drop database test;
Query OK, 0 rows affected (0.07 sec)

Remember from this point all commands that create or change anything in MySQL must now be run on the master server. With this in mind, create a read only user, you don’t want your websites accidentally writing to the slaves, that will break replication!

mysql> grant select on *.* to 'selectuser'@'%' identified by 'password';
Query OK, 0 rows affected (0.00 sec)

mysql> grant select on *.* to 'selectuser'@'localhost' identified by 'password';
Query OK, 0 rows affected (0.00 sec)

And then obivously, a write user:

mysql> grant select,insert,update,delete on *.* to 'writeuser'@'%' identified by 'password';
Query OK, 0 rows affected (0.00 sec)

mysql> grant select,insert,update,delete on *.* to 'writeuser'@'localhost' identified by 'password';
Query OK, 0 rows affected (0.01 sec)

Proxy Servers

Repeat this whole section on both proxy servers (bhlabs1/6)
Install the proxy:

# apt-get updatep
# apt-get install mysql-proxy

Initial startup:

# mysql-proxy --admin-address=:1337 --proxy-backend-addresses=192.168.11.36:3306 --proxy-backend-addresses=192.168.11.37:3306 --proxy-backend-addresses=192.168.11.38:3306 &

Check the proxy is running via netstat to see if the ports have been bound:

# netstat -l
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 *:4040                  *:*                     LISTEN
tcp        0      0 *:1337                  *:*                     LISTEN

Port 4040 is the proxy inbound port
Port 1337 is the admin database

Connect to admin database:

# mysql -h 127.0.0.1 -P 1337 -u root -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.1.20-agent MySQL Enterprise Agent

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql>

And then check your proxy_config tables at it should list your backend servers:

mysql> select * from proxy_config;
+----------------------------+--------------------+
| option                     | value              |
+----------------------------+--------------------+
| admin.address              | :1337              |
| proxy.address              | :4040              |
| proxy.lua_script           | NULL               |
| proxy.backend_addresses[0] | 192.168.11.36:3306 |
| proxy.backend_addresses[1] | 192.168.11.37:3306 |
| proxy.backend_addresses[2] | 192.168.11.38:3306 |
| proxy.fix_bug_25371        | 0                  |
| proxy.profiling            | 1                  |
+----------------------------+--------------------+
8 rows in set (0.00 sec)

Quit out of mysql and then reconnect on the proxy ingress port:

# mysql -h 127.0.0.1 -P 4040 -u selectuser -p

And you should then be able to spot a Sleep connection on one of your slave servers through:

mysql> show full processlist;
+----+------------+---------------------+------+---------+------+-------+-----------------------+
| Id | User       | Host                | db   | Command | Time | State | Info                  |
+----+------------+---------------------+------+---------+------+-------+-----------------------+
| 20 | selectuser | 192.168.11.30:58522 | NULL | Sleep   |  161 |       | NULL                  |
| 21 | selectuser | localhost           | NULL | Query   |    0 | NULL  | show full processlist |
+----+------------+---------------------+------+---------+------+-------+-----------------------+
2 rows in set (0.00 sec)

Success!

You can now test failover by shutting down one of your slaves, running queries, even inserts (on the master!), and then starting the slave again. The slave will know the last log position it was at on the master logs and be able to catch back up once you restart it. The proxy will not route queries to a server thats down.

High Availability (Optional)

This will allow your proxy servers to failover between each other on the same address using heartbeat. For how to setup heartbeat initially see the guide here, we’re going to pick up at the end of that guide and just join our mysql-proxy to that shared IP address.

All you need to do to get it to listen on the shared address is add 1 more switch to the startup command, proxy-address:

# mysql-proxy --proxy-address=192.168.11.40:4040 --admin-address=:1337 --proxy-backend-addresses=192.168.11.36:3306 --proxy-backend-addresses=192.168.11.37:3306 --proxy-backend-addresses=192.168.11.38:3306 &

You can then check it’s listening on the specified IP via netstat again. Remember if heartbeat hasn’t failed over to this machine then you won’t be able to reach this address!

# netstat -l
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 192.168.11.40:4040      *:*                     LISTEN

Furthur Reading

Automatically routing writes and reads

It is possible to use 1 user for both your reads and your writes through 1 IP/Port by using mysql-proxy’s very extensive LUA support to route queries to different servers based on queries. See here and here for examples.

Other funky LUA hacks

There’s plenty you can do with LUA and mysql-proxy. O’Reilly published a good guide with examples such as rewriting queries on the fly to correct typos, executing remote shell commands through MySQL, etc.  Worth a read if you want to do anything automated with it.

I wrote this for a PHP/MySQL site that needed to have it’s databases backed up hourly and the whole site including uploads backed up daily. They wanted all databases going back 24 hours every hour, and then all content going back 7 days every day.  This creates a directory structure that looks like the following:

In this scenario assume the following:

• /mnt/backup/servername is an NFS mounted location I am backing up to
• /var/www is the location where the website is stored

Firstly for the impatient, here is the 2 lines you need to put into crontab:

0 0 * * * mkdir -p /mnt/backup/servername/`date +\%A`;  mysqldump -uUser -pPassword --all-databases > /mnt/backup/servername/`date +\%A`/all-databases-daily.sql; tar -czf /mnt/backup/servername/`date +\%A`/www.tar.gz /var/www/
0 * * * * mysqldump -uUser -pPassword --all-databases > /mnt/backup/servername/all-databases-hourly-`date +\%H`.sql

For the less impatient I’ll explain how it works.

Daily Backups

I created 2 cronjobs. The first is to do the daily backups. Lets break this job down:

First of all we want the job to run every day at midnight

0 0 * * *
^----------Run at minute 0
  ^--------Run at hour 0
    ^------Run every day of month
      ^----Run every month of year
        ^--Run every day of week

Next we need to check our backup target directory exists and is named correctly:

mkdir -p /mnt/backup/servername/`date +\%A`;
  ^------------------------------------------Create directory
       ^-------------------------------------Create recursively, so if /mnt doesn't exist (god forbit!) it will create it as well
         ^---------------------^-------------Root of backup folder
                                ^---------^--Output the day of the week in English (eg "Saturday")

Next we need to dump the databases into the just created folder:

mysqldump -uUser -pPassword --all-databases > /mnt/backup/servername/`date +\%A`/all-databases-daily.sql;
   ^------------------------------------------------------------------------------------------------------MySQL cli tool used to dump content to stdout
          ^---------------^-------------------------------------------------------------------------------Your username and password
                                   ^----------------------------------------------------------------------This switch tells mysqldump to output every database in the server the user can access
                                            ^-------------------------------------------------------------Using > diverts stdout to a given file
                                              ^--------------------------------^--------------------------Backup folder root and date as above example
                                                                                        ^-----------------Target filename

And finally, we tar and gzip the web root for the daily backup into the same folder:

tar -czf /mnt/backup/servername/`date +\%A`/www.tar.gz /var/www/
^------^---------------------------------------------------------Create a new archive using gzip and output to a file
         ^-------------------------------------------^-----------File to output
                                                       ^-------^-Directory to archive

Hourly Backups

First the frequency:

0 * * * *
^----------Run at minute 0
  ^--------Run every hour
    ^------Run every day of month
      ^----Run every month of year
        ^--Run every day of week

And then the database dumps themselves:

mysqldump -uUser -pPassword --all-databases > /mnt/backup/servername/all-databases-hourly-`date +\%H`.sql
   ^------------------------------------------------------------------------------------------------------MySQL cli tool used to dump content to stdout
          ^---------------^-------------------------------------------------------------------------------Your username and password
                                   ^----------------------------------------------------------------------This switch tells mysqldump to output every database in the server the user can access
                                            ^-------------------------------------------------------------Using > diverts stdout to a given file
                                              ^--------------------------------^--------------------------Backup folder root and date. This time using %H instead of %A tells it to output the hour
                                                                                        ^-----------------Target filename

Looks like…

The final output, after it’s run for a week, should look something like this:

root@localhost:~# ls -l /mnt/backup/servername/
total 33216
-rw-r--r-- 1 768 500 1416355 2009-05-08 23:38 all-databases-hourly-00.sql
-rw-r--r-- 1 768 500 1416355 2009-05-09 00:38 all-databases-hourly-01.sql
-rw-r--r-- 1 768 500 1416355 2009-05-09 01:38 all-databases-hourly-02.sql
-rw-r--r-- 1 768 500 1416355 2009-05-09 02:38 all-databases-hourly-03.sql
-rw-r--r-- 1 768 500 1416355 2009-05-09 03:38 all-databases-hourly-04.sql
-rw-r--r-- 1 768 500 1416355 2009-05-09 04:38 all-databases-hourly-05.sql
-rw-r--r-- 1 768 500 1416355 2009-05-09 05:38 all-databases-hourly-06.sql
-rw-r--r-- 1 768 500 1416355 2009-05-09 06:38 all-databases-hourly-07.sql
-rw-r--r-- 1 768 500 1416355 2009-05-09 07:38 all-databases-hourly-08.sql
-rw-r--r-- 1 768 500 1416355 2009-05-09 08:38 all-databases-hourly-09.sql
-rw-r--r-- 1 768 500 1416355 2009-05-09 09:38 all-databases-hourly-10.sql
-rw-r--r-- 1 768 500 1416355 2009-05-09 10:38 all-databases-hourly-11.sql
-rw-r--r-- 1 768 500 1416355 2009-05-09 11:38 all-databases-hourly-12.sql
-rw-r--r-- 1 768 500 1416457 2009-05-08 12:38 all-databases-hourly-13.sql
-rw-r--r-- 1 768 500 1416457 2009-05-08 13:38 all-databases-hourly-14.sql
-rw-r--r-- 1 768 500 1416383 2009-05-08 14:38 all-databases-hourly-15.sql
-rw-r--r-- 1 768 500 1416383 2009-05-08 15:38 all-databases-hourly-16.sql
-rw-r--r-- 1 768 500 1416355 2009-05-08 16:38 all-databases-hourly-17.sql
-rw-r--r-- 1 768 500 1416355 2009-05-08 17:38 all-databases-hourly-18.sql
-rw-r--r-- 1 768 500 1416355 2009-05-08 18:38 all-databases-hourly-19.sql
-rw-r--r-- 1 768 500 1416355 2009-05-08 19:38 all-databases-hourly-20.sql
-rw-r--r-- 1 768 500 1416355 2009-05-08 20:38 all-databases-hourly-21.sql
-rw-r--r-- 1 768 500 1416355 2009-05-08 21:38 all-databases-hourly-22.sql
-rw-r--r-- 1 768 500 1416355 2009-05-08 22:38 all-databases-hourly-23.sql
drwxr-xr-x 2 768 500      54 2009-03-05 23:41 Friday
drwxr-xr-x 2 768 500      54 2009-03-08 23:41 Monday
drwxr-xr-x 2 768 500      54 2009-03-06 23:41 Saturday
drwxr-xr-x 2 768 500      54 2009-03-07 23:41 Sunday
drwxr-xr-x 2 768 500      54 2009-03-04 23:41 Thursday
drwxr-xr-x 2 768 500      54 2009-03-03 16:21 Tuesday
drwxr-xr-x 2 768 500      54 2009-03-03 23:41 Wednesday
root@localhost:~# ls -l /mnt/backup/servername/Friday/
total 357600
-rw-r--r-- 1 768 500   1414721 2009-05-07 23:38 all-databases-daily.sql
-rw-r--r-- 1 768 500 364579398 2009-05-08 00:30 home.tar.gz

Getting PHP and oracle to play nice is a massive pain, so this post is more a log for myself next time I have to do it more than anything else!

The procedure requires you to compile PHP from source, so if thats not your thing or your easily scared don’t bother continuing!

Firstly an enormous yum install command to get everything we need in one go:

# yum install openssh-server httpd httpd-devel apxs libx openssl-devel curl-devel libpng-devel\
 oci8-devel libxml libxml2 libxml2-devel libjpeg libXpm libXpm-devel libpng libxslt libxslt-devel\
 freetype freetype-devel

Go and grab the instant client from the oracle website and install it with RPM:

# rpm -i oracle-instantclient11.1-basic-11.1.0.7.0-1.i386.rpm
# rpm -i oracle-instantclient11.1-devel-11.1.0.7.0-1.i386.rpm

And then get the PHP source (substitute for whatever the latest version is at your compile time obviously):

# wget http://uk.php.net/get/php-5.2.8.tar.gz/from/this/mirror
# tar -xzf php-5.2.8.tar.gz
# cd php-5.2.8

And finally configure it all together:

# ./configure  --with-apxs2=/usr/sbin/apxs --with-zlib --without-gdbm --enable-force-cgi-redirect\
 --enable-ftp --enable-xml --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin\
 --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib\
 --libexecdir=/usr/libexec --localstatedir=/var --mandir=/usr/share/man --infodir=/usr/share/info\
 --with-config-file-path=/etc --with-config-file-scan-dir=/etc/php.d --with-pear --enable-pdo\
 --with-oci8=instantclient,/usr/lib/oracle/11.1/client/lib --with-pdo-oci=instantclient,/usr,11.1\
 --with-xsl --with-curl --with-openssl --enable-soap --enable-sockets --enable-sigchild --enable-xdebug\
 --with-jpeg-dir --with-xpm-dir --with-png-dir --with-freetype-dir --with-ttf --enable-gd-native-ttf --with-gd

And with a bit of luck you should be up and running. Obviously now you have to do battle with Oracle itself which is a minefield of confusion!

I wanted to find a simple to maintain and expand soultion for load balancing a web cluster with high availability. I have found my solution in HAProxy.

Scenario

Bluhalo Labs HAProxy Test Rig

This demo scenario is in the following enviroment:

• Network Configuration
  • Network: 192.168.11.0
  • Subnet Mask: 255.255.255.0
  • Gateway: 192.168.11.254 (Although this is irrelavant)
• 1 x Shared IP for the Load Balancers
  • 192.168.11.40
• 2 x Load Balancers
  • BHLabs1 – 192.168.11.30
  • BHLabs6 – 192.168.11.39
• 4 x Web Servers
  • BHLabs2 – 192.168.11.35
  • BHLabs3 – 192.168.11.36
  • BHLabs4 – 192.168.11.37
  • BHLabs5 – 192.168.11.38

At the start of this setup all machines are running Ubuntu 8.04 Server from a standard install with openssh-server installed and the root password set. All setup commands are run as root or with sudo.

Web Server Configuration

Basic Config

As we are only doing this as a basic test, a very simple Apache config is required.
This will install Apache 2 and also PHP5 to give us some basic scripting to output server name for testing etc that you may wish to play with later.

# apt-get -y install php5

Next you need to create a check file for HAProxy to look for from the load balancers. This file will be used to determine if the servers are up. This will create a blank file called check.txt in the default DocumentRoot for Apache.

# touch /var/www/check.txt

Now stick your test index.html in that directory as well.

# echo "oh hi" > /var/www/index.html

Log Modification

Filtering out HAProxy health checks

You don’t want to log hits to the check.txt file in your Apache logs, so put an exclusion in your VirtualHost directive. here’s an example of how:

<VirtualHost *>
  ServerAdmin server-alert@bluhalo.com
  DocumentRoot /var/www/
  ErrorLog /var/log/apache2/error.log
  LogLevel warn
  CustomLog /var/log/apache2/access.log combined env=!dontlog
  SetEnvIf Request_URI "^/check\.txt$" dontlog
</VirtualHost>

Modify the access logs

HAProxy will act as a completely transparent proxy so by default the web servers will log the load balancers IP in it’s logs instead of the user’s. HAProxy add’s the user’s IP to the header in the “X-Forwarded-For” field, so you need to modify the log configuration in your apache2.conf to take advantage of this:

vi /etc/apache2/apache2.conf

Search for entries that start “LogFormat” …

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent

… and swap “%h” for “%{X-Forwarded-For}i” like:

LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent

Load Balancer Configuration

You need to install HAProxy and Heartbeat for this setup to work. HAProxy provides your load balancing functionality and Heartbeat provides your high-availability failover functionality.

# apt-get -y install haproxy heartbeat-2

Let’s start with HAProxy as thats the easier one :)

HAProxy

Open up the HAProxy config file:

# vi /etc/haproxy.cfg

and replace the whole file on both servers with the following:

global
  log 127.0.0.1 local0
  log 127.0.0.1 local1 notice
  maxconn 4096
  #debug
  #quiet
  user haproxy
  group haproxy

defaults
  log global
  mode  http
  option  httplog
  option  dontlognull
  retries 3
  redispatch
  maxconn 2000
  contimeout  5000
  clitimeout  50000
  srvtimeout  50000

listen bhlabslb 192.168.11.40:80
  mode http
  stats enable
  stats auth admin:password
  balance roundrobin
  option httpclose
  option forwardfor
  option httpchk HEAD /check.txt HTTP/1.0
  server  inst1 192.168.11.35:80 cookie server01 check inter 2000 fall 3
  server  inst2 192.168.11.36:80 cookie server02 check inter 2000 fall 3
  server  inst3 192.168.11.37:80 cookie server01 check inter 2000 fall 3
  server  inst4 192.168.11.38:80 cookie server02 check inter 2000 fall 3
  capture cookie vgnvisitor= len 32

  rspidel ^Set-cookie:\ IP= # do not let this cookie tell our internal IP address

You also have to allow the HAProxy service to start. Change the ENABLED value in the /etc/default/haproxy:

# Set ENABLED to 1 if you want the init script to start haproxy.
ENABLED=1
# Add extra flags here.
#EXTRAOPTS="-de -m 16"

Heartbeat

We will be using Heartbeat to pass the shared IP address (192.168.11.40) between our 2 load balancers if one goes down. To do this, it needs to be able to bind to an address that doesn’t yet exists on the system. In order to allow this you need to add the following to /etc/sysctl.conf:

# Allow HAProxy shared IP
net.ipv4.ip_nonlocal_bind = 1

and then run:

# sysctl -p

Heartbeat requires 3 main configuration files which do not come with the install. First of all the authkey. Do the following on both servers:

# vi /etc/ha.d/authkeys

add the following content, making sure you replace MyPassword with a secure string. This needs to be the same on both servers:

auth 3
3 md5 MyPassword

This file MUST be accessible only by root or Heartbeat won’t start:

# chmod 600 /etc/ha.d/authkeys

Next on each server create the following. Run:

# uname -n

to get the kernels take on the local hostname, and then insert this into:

# vi /etc/ha.d/haresources

in the following syntax:

BHLabs1 192.168.11.40

and

BHLabs6 192.168.11.40

Note the hostname changes but not the IP. The IP is the shared IP. Finally the main Heartbeat config file:

vi /etc/ha.d/ha.cf

On the first server:

#
#       keepalive: how many seconds between heartbeats
#
keepalive 2
#
#       deadtime: seconds-to-declare-host-dead
#
deadtime 10
#
#       What UDP port to use for udp or ppp-udp communication?
#
udpport        694
bcast  eth0
mcast eth0 225.0.0.1 694 1 0
ucast eth0 192.168.11.30
#       What interfaces to heartbeat over?
udp     eth0
#
#       Facility to use for syslog()/logger (alternative to log/debugfile)
#
logfacility     local0
#
#       Tell what machines are in the cluster
#       node    nodename ...    -- must match uname -n
node    BHLabs1
node    BHLabs6

and on the second server:

#
#       keepalive: how many seconds between heartbeats
#
keepalive 2
#
#       deadtime: seconds-to-declare-host-dead
#
deadtime 10
#
#       What UDP port to use for udp or ppp-udp communication?
#
udpport        694
bcast  eth0
mcast eth0 225.0.0.1 694 1 0
ucast eth0 192.168.11.39
#       What interfaces to heartbeat over?
udp     eth0
#
#       Facility to use for syslog()/logger (alternative to log/debugfile)
#
logfacility     local0
#
#       Tell what machines are in the cluster
#       node    nodename ...    -- must match uname -n
node    BHLabs1
node    BHLabs6

Testing

Restart all the services. On the web servers run:

# apache2ctl restart

and on the load balancers:

# /etc/init.d/heartbeat start
# /etc/init.d/haproxy start

You should then be able to hit http://192.168.11.40 and see your test webpage!
You should also have a page full of stats to please they eyes from HAProxy at http://192.168.11.40:81/haproxy?stats. This can be turned off by removing the following 2 lines from the haproxy.cfg. This should be removed in a production enviroment.:

 stats enable
 stats auth admin:password

Try the following:

• Kill the heartbeat service on each load balancer in turn while running “watch ifconfig”. You should see the IP address move from server to server.
• Pull the plug on either of the load balancers while running a ping from your PC to 192.168.11.40, the ping should never fail.
• Shut down Apache on any of the web servers, you should see them go red in the stats page within 2 seconds and they will be removed from the load balanced group until they come back up.

Caveats and best practices

This setup is lacking in some important best practices:

1. The heartbeat should run on it’s own network, usually a VLAN dedicated to it.
2. The web servers should sit on a different network to the frontend on the load balancers. The load balancers would then have a second interface out to that network to connect to the web servers.

Later I will follow this up with a way to add MySQL to this configuration.

Reference

• HAProxy documentation: http://haproxy.1wt.eu/download/1.3/doc/configuration.txt
• Heartbeat documentation: http://www.linux-ha.org/LearningAboutHeartbeat

So how does the current generation of Browsers fair on the current generation of Acid test?

Safari 4 a clear winner! I’m supprised about Internet Explorer 8, being the newest of the bunch. I had high  hopes for them at least trying in this release!

If you’re used to working in a nice IDE or some kind of GUI when programming, this may be of interest to you.

The common complaint I hear about when people have to work from the command line such as on remote servers is that there is no nice editor. To me this sounds ludicrous just because I’m used to using Vi all day, but I can see how it would be a problem to those used to a prettier interface

Insert the following lines into a .vimrc file in your home directory: ($ vi ~/.vimrc)

:syntax on
:set expandtab
:set tabstop=2
:set shiftwidth=2
:set showtabline=2

This will do the following for you:

• Enabled coloured syntax highlighting for you while editing, based on language
• Tell Vi to use the equivalent number of spaces instead of tabs when you hit the Tab key
• Set the number of spaces inserted to 2
• Set the number of spaces inserted when indenting to 2
• Force the tab bar to be shown along the top all the time

Here we use double spacing instead of Tabs in line with the Symfony standards. You can adjust the values above to fit whatever your coding standards are.

You can navigate between tabs by using:
:tabn & :tabp:
open new tabs with:
:tabe
and open files into a new tab with:
:tabe filename
When you’re at the command line, you can open multiple files at once into multiple tabs by doing:
$ vi file1.php file2.php -p

Finally, if you are a windows user and use PuTTY to connect to servers over SSH, you should definitly override the default colors. Right click the task bar of your open PuTTY session, go to the colours section, and check the box for “Use system colours”. This makes the background white instead of black, and stops you having to squint to make out the annoying blue against black you get when working remotely.

OK so me and my housemate have a problem. We have accumulated to much stuff over the years between us, we’ve pretty much run out of solutions to hold it all without buying an enterprise style SAN, but keeping it all in the same volume. Obviously this is to expensive for a home user! We’re looking at finding a home for ~3.5 terabytes of data, including music, backups, films, TV, iTunes mounts etc. I’ve decided to post this on here because it’s also useful to SOHO and medium business who need a storage solution.

We looked at various methods of achieving loads of storage, but have settled on a stand alone NAS. The main reason for this is power consumption; we calculated having a server running with a load of hard disks was going to cost us £30 per month whereas a NAS would only cost us £4 per month!

These are the solutions we rulled out:

• OpenFiler – Way to restrictive on it’s own with sharing and user management
• OpenFiler + iSCSI + Windows 2003 Server – Gets around the restriction above, but costs a fortune having to run 2 servers!
• Windows 2003 Software RAID alone – Unstable, and slow. We actually ran with this for a while but it kept resynching which would take transfer speed down to <6Mb/s. Also there is no expansion option!
• Windows 2003 with a motherboard RAID solution – Most of these come down to runing it off a motherboard unless you spend silly money. On board RAID was usually fine until you wanted to change anything. You can’t move the disks around, it’s slow to rebuild, you can’t expand, etc etc. You are also limited to the number of ports on your motherboard.
• Windows 2003 with PCI-E RAID Card – We could get a seperate RAID card but there is really only 2 SATA2 RAID cards out there and they cost a fortune. You can get cheap SCSI cards, but SCSI disks are both expensive and small capacity.

So we settled on the fact it’s going to be easiest, most economic, and most reliable to buy a standalone NAS appliance. This however throws up the question: which one! There are loads on the market, so we trawled the web and found the most popular, and drew up the following chart to compare them all (PDF version beneath). We are torn between the QNAP TS-509 with 1TB disks and TS-409 with 1.5TB disks. That’s up to next months bank balance I guess!

NAS Comparison Chart

(You can download as a PDF here)

Amazon have launched Amazon Cloudfront today with an announcment on their blog as a service to layer on top of S3. Cloudfront allows you to serve content through Amazon’s edge cache which is located around the world in key locations. When you request content through the assigned domain name, it automatically routes you to the physically closest location to retrieve content. If the content is not yet there, Cloudfront will fetch it from S3 and cache it.

One of the most important services this provides is Asian based distribution points for content on S3; previously only available in Europe or USA. This should increase uptake of AWS in Asia, and with a larger uptake hopefully Amazon will look to make other services worldwide such as DevPay and mechanical Turk (currently only available in the USA).

For a full description including the list of locations Cloudfront distributes through take a look at the detailed description on the official Cloudfront page.

Scenario: You’re new to the cloud infrastructure market but like the idea of flexible billing, rapid rollout (or termination) zero commitments and the possibility for inifite expansion.

These four blog articles all posted today take you through the basics of understanding what you need to get started in a non-vendor biased way:

First, is it right for you? Of course it probabaly is, but this explains why:

http://bob-jansen.nl/?p=67

Next, how do you identify what makes a cloud providor a cloud providor?

http://www.redmonk.com/jgovernor/2008/11/12/15-ways-i-am-wrong-about-enterprise-cloud-computing/

Then, how do you choose which provider to us? Some are limited to certain technologies, whereas some are so open you can treat your cloud instance however you like.

http://www.manyniches.com/cloudcomputing/cloud-strategy-a-question-of-motivations/

And finally, how do you pitch to your customer that it’s safe? The traditional idea of “this website is on this server” is a hard idea to shake for some people and this could make them doubt the reliability or stability of something that doesn’t physically exist. This blog post covers SLAs in cloud computing:

http://www.technewsworld.com/story/Cloud-Computing-Part-3-SLA-Spirit-in-the-Sky-65137.html

Cloud computing is such a hot topic right now that with the most minimal of effort you can find all the resources, contacts and knowledge you need to get started quickly with cloud computing. A quick search in twitter for example produces a hive of chit chat on the topic with relevant links, experiences and tutorials.

Operating System Usage Prediction

These figures are extrapolated from W3Schools.com. I have predicted data for the next 4 years with a linear scale based on the last 5 years. The vertical axis has a logarithmic scale so the OSX and Linux data is meaningful. The graph shows a very slight decline in Windows usage and a notable increase in Apple OSX and Linux usage.

Obviously there are plenty of unforseen changes that could have an effect on this graph. I see the move towards OSX from Windows in the mainstream as inevitable, and I would imagine the change is more likely to be a sudden surge over a period of months rather than a smooth decline of Windows usage over a number of years.

It is interesting to compare the initial Vista predictions made by Steven York just before Vista’s release with the real stats in hindsight. The obviously low take up of Vista leaves most users on Windows XP, which when you compare to the look and feel of modern environments (for example: Websites, Mobile Phones, Latest Apple OS) it’s very clunky and unintuitive.