HAProxy load balancer testing
I needed to look into an appropriate solution to prevent requests being dropped when adding new nodes to my HAProxy load balancer. Initially I thought that by upgrading to a later version (1.4.9) I could take advantage of a `reload` function within the init script. Something that the community has been screaming out for a long time.
After installing HAProxy, I checked the init script and could see the function, all be it looked like the right one and following the code I could see that it would store connections and load a new binary.
02 | $exec -c -q -f /etc/$prog/$prog.cfg |
04 | echo "Errors in configuration file, check with $prog check." |
07 | echo -n $"Reloading $prog: " |
08 | $exec -D -f /etc/$prog/$prog.cfg -p /var/run/$prog.pid -sf $(cat /var/run/$prog.pid) |
The interesting portion of the init script function is `$exec -D -f /etc/$prog/$prog.cfg -p /var/run/$prog.pid -sf $(cat /var/run/$prog.pid)` which reloads the config file and preserves the existing PID number waiting for it finish with connections and therefore kill when complete. Well that's what I thought, looks like I was wrong.
I fired up a machine I have with HTTPerf installed and ran some connection tests without issuing any init controls to get a feel for how my load balancer stacks up. Not a problem, the load balancer easily took 1000 requests per second without bottlenecking which shows that my plan to implement HAProxy wouldn't slow down my application stack.
1 | httperf --hog --timeout=5 --client=0/1 --send-buffer=4096 --recv-buffer=16384 --num-conns=100000 --num-calls=1 --uri=/<url_behind_lb> --server=<loadbalancer_ip> --port=80 --rate=1000 -v |
Now to try the tests whilst issuing a `reload` from the load balancer. I fired up the HTTPerf test and tried again, waiting 2 seconds before issuing the `reload` (I used a script to make this fair). The results were not conclusive, but they did indicate that requests were being given HTTP 5xx which breaks my application stack. Effectively this is giving users a 503 Timeout, which is something I really don't want to do.
I decided to scale down the requests and try the tests again and also received the same results. Below are my findings of `reload` verses `restart`.
|
HaProxy (v1.4.19)
|
Reload
|
|
|
|
Restart
|
|
|
|
|
|
|
Result 1
|
Result 2
|
Result 3
|
|
Result 1
|
Result 2
|
Result 3
|
|
|
|
|
|
|
|
|
|
|
|
Test 1: 10 connections per second
|
|
0
|
0
|
0
|
|
1
|
2
|
1
|
|
|
|
|
|
|
|
|
|
|
|
Test 2: 50 connections per second
|
|
5
|
0
|
0
|
|
8
|
10
|
9
|
|
|
|
|
|
|
|
|
|
|
|
Test 3: 100 connections per second
|
|
1
|
0
|
9
|
|
12
|
12
|
16
|
|
|
|
|
|
|
|
|
|
|
|
Test 4: 200 connections per second
|
|
2
|
8
|
0
|
|
25
|
33
|
59
|
|
|
|
|
|
|
|
|
|
|
|
Test 5: 400 connections per second
|
|
0
|
4
|
0
|
|
96
|
86
|
62
|
To sum up, I'm going to have to investigate the uses of Nginx as a suitable load balancer to overcome this issue which we all hoped had been resolved…. Unless I'm missing something? Who knows? I'll find out and get to the bottom of it.
About The Ninja
32 year old Carl hails from London but lives in Staffordshire, working for a specialist company that provider hosting and search services for the automotive industry as a Linux Systems Administrator and Cloud Engineer.
He likes to dabble in PHP, Bash and anything geeky.