When tweaking performance settings in Apache…

This post is to do with a discovery I made while making performance tweaks on the webservers for http://www.tottenhamhotspur.com. This particular setup has six loadbalanced webservers and eight database back-ends designed to cater for a high amount of shorter bursts of traffic.

There was  problem however with the site not responding, even though the servers were under a tiny load, and the load balancer was not under much stress. I could connect to localhost just fine (telnet is a magnificent troubleshooting tool, is it not?). Running `ps ax|grep httpd | wc -l` I counted exactly 256 threads running for Apache. So I went and edited httpd.conf and raised ServerLimit and MaxClients to 1024 (from 256). I then restarted the service with apachectl graceful. Went back and checked during the next days traffic, and it was still not responding. When looking at the error log:

WARNING: MaxClients of 1024 exceeds ServerLimit value of 256 servers, lowering MaxClients to 256. To increase, please see the ServerLimit directive.

This confused me as I had changed both values, and confirmed this with cat on the conf file. it appeared the ServerLimit value was being ignored. I tried an apachectl restart in case graceful was leaving zombies, but that made no difference either. It turns out, that you have to do a full graceful-stop and start on Apache in order for it to pick up the change in ServerLimit. Annoying?!