Synergi tips and tricks

The joy of deploying a split world DNS system with views.

Read...

Imagine the following situation:

• 2 load balanced apache frontend in pre-fork with dozens of site.
• 2 app server running some code based on tomcat
• Apache configured with mod_proxy_balancer to proxy some queries to the backend using ajp13

Under a light to moderate load, everything runs smoothly. At 15 req/s it's still runs smoothly. Suddenly, without more traffic, everything stalls. The number of apache workers instantly climbs to 256 (the MaxClients you've set up) and your load balancer can't get a sign of life from your web server and decide to kick it out of the pool.

After cursing and blaming the app to take too much time answering and crippling the system now I have a fairly good view of what happened. As the traffic increase (even for unrelated sites), more and more apache children were launched. Because we run the pre-fork model (the default under *nix), each child will open it's own ajp connection to the backend as it needs it. When that backend reaches it's maxProcessors, the connections starts the go into the backlog of tomcat. But because the answer doesn't come straight away, it keeps the apache child "in use" for that request, which trigger the creation of more and more apache child processes which exacerbate the issue. The same effect could have been caused by a backend responding just moderately slow (15r/s and 15s to answer, that's 225 extra slots used).

Congratulation, with just a butterfly wing flap in one corner, you managed to get the whole building collapse. And whatever the requests per second goes back to, even less than one per sec will keep your system on it's knees.

Now let's look at options to mitigate the effect of the butterfly:

• be sure that tomcat.maxProcessors >= sum(apache.maxclients)
• have apache.ProxyTimeout set to something low enough (default is 300). 10 or 20 should be enough.
• try to keep your apps within a small execution timeframe, or raise an exception and lemming it if it takes too long.

I could plagiarise Brian Moore Despite the tons of examples and docs, mod_proxy_balancer is voodoo. Damned cool voodoo, but still voodoo.

Today I had to move users/developers and their working environment to a new box. How to go from static to dynamic.

Read...

How to break a fully working mysql master/master cluster for some changes on full text search and stopwords.

Read...