Performance Tuning of HAProxy for Database Load Balancing - Slides

Powering Your Uptime
HAProxy Technologies
HAProxy and Mysql
EMEA Headquarters
3, rue du petit Robinson
ZAC des Metz
78350 Jouy en Josas
France
http://www.haproxy.com
HAProxy Technologies - HAProxy and Mysql - http://www.haproxy.com/ - Private document

Performance Tuning of HAProxy for Database
Load Balancing

Agenda
• Introduction of HAProxy Technologies
• Inside HAProxy
• New features in HAProxy 1.5 + focus on SSL
• HAProxy multi-process: advantages, limitations, configuration example
• Dynamic re-configuration
• What can HAProxy tell you about your application and your database
• Weakness in MySQL client library
• Hints for short live connections
• Hints for persistent connections
• HAProxy active/active failover setups
• Security considerations

HAProxy Technologies, who are we???
• 8 years old company, used to be named exceliance
• HAProxy task force: main developers, main contributors
• HAProxy pure player:
• ALOHA Load-Balancer: the HAProxy appliance
• HAPee: improved and supported HAProxy package
• Prof services: expertise on your HAProxy deployments
• More to come, so stay tuned!!!!!
• serve customers all over the world
• Website: http://www.haproxy.com/
• Twitter: https://twitter.com/haproxy_tech

Inside HAProxy
• How is HAProxy implemented:
• userland software
• single thread / process, 100% event driven, non blocking, priority based internal scheduler
• HTTP / TCP reverse proxy, relies on underlying kernel to manage TCP connections
• forward streams between two connections. Can't forward datagram (UDP)
• get client traffic in a frontend and forward it to servers through a backend
• Startup / reload procedure:
1. load, parse, and validate all configuration files
2. signal previous instance (if any) that it must release listening ports
3. bind new listening ports to accept incoming traffic
4. upon success, the reload command decides whether previous instance continues to
manage existing connections or immediately quits
5. upon failure, new instance stops and previous one continues its job
• Note: the previous instance will be called the dying process in the next slides

Inside HAProxy - the dying process
• This is the previous process remaining after a configuration reload.
• A few things to know about the dying process:
• it handles established connections
• it runs the old configuration
• can't be managed by the stats socket
• does not perform any health check

HAProxy 1.5 new features
• SSL: one of the most complete stack out there (https://istlsfastyet.com/)
• full IPv6: both client and server side, internal features
• support of UNIX sockets
• HTTP compression
• HTTP keep alive (!!)
• maps
• agent check
• raw TCP check
• improved stick tables: behavior analysis
• many new ACLs
• better management of multi process
• custom log format
• etc...

HAProxy 1.5: A focus on SSL
• requires OpenSSL
• many advanced features:
• TLS/SSL version and cipher choice
• cipher traffic on client or server side
• client certificate management
• SNI: SNI based routing or virtual hosting
• NPN/ALPN
• SSL information in HTTP headers
• AES-NI CPU instructions (with compatible library)
• multi-process session cache
• haproxy -vv output when built with SSL support:
Built with OpenSSL version : OpenSSL 1.0.1i 6 Aug 2014
Running on OpenSSL version : OpenSSL 1.0.1i 6 Aug 2014
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Special thanks to @emericBr

Any good reason to switch to 1.5 for MySQL ??
Don't fix it if it works!
• No, since 1.4 has already the most interesting basic features:
• TCP splicing for long live session
• mysql health checks
• connection queueing for short live sessions
• on-marked-down shutdown-sessions
• A few advantages for 1.5:
• improve multiprocess mode (next release will carry on improving this point)
• raw tcp-check to execute arbitrary send/expect binary checks (not tested with MySQL)
• agent check: an agent installed on the server can update server's weight in HAProxy
• on-marked-up shutdown-backup-sessions
• painless upgrade, but test in lab first...
• Note: SSL offloading can't work with MySQL. MySQL SSL is a bit like IMAP's STARTTLS

HAProxy multi-process: advantages
• ability to dedicate a process to a task (or application, or protocol)
In example: 1 process for HTTP and 1 process for MySQL
• scale up: same hardware, more processing capacity by binding processes to different CPU cores
• useful when massive SSL offloading processing is required
key generation scales almost linearly with number of processes, but session resumption gets
little gain over 3 processes

HAProxy multi-process: limitations
Each process has its own memory area, which means:
• debug mode cancels multi-process (a single process is started)
• frontend(s) and associated backend(s) must run on the same process
• not compatible with peers section (stick table synchronization)
• information is stored locally in each process memory area and can't be shared:
• stick table + tracked counters
• statistics
• maxconn (queue management)
• connection rate
• Each HAProxy process performs its health check:
• a service is probed by each process
• a service can temporarly have different status in each process
• managing a configuration which starts up multiple processes can be more complicated

HAProxy multi-process: configuration example
1 # **DON'T RUN IN PRODUCTION, THERE ARE NO TIMEOUTS**
2 global
3 nbproc 2
4 cpu-map 1 1
5 cpu-map 2 2
6 stats socket /var/run/haproxy/socket_web process 1
7 stats socket /var/run/haproxy/socket_mysql process 2
8
9 defaults HTTP
10 bind-process 1
11 mode http
12
13 frontend f_web
14 bind 127.0.0.1:9000
15 default_backend b_web
16
17 backend b_web
18 server s 127.0.0.1:8000
19
20 defaults MYSQL
21 bind-process 2
22 mode tcp
23
24 frontend f_mysql
25 bind 127.0.0.1:3306
26 default_backend b_mysql
27
28 backend b_mysql
29 server s 192.168.10.11:3306

HAProxy dynamic configuration
• As seen before, HAProxy reads and loads its configuration at startup from a raw text file
• It is possible to update some settings from HAProxy's in memory configuration
• don't forget to update to update the configuration text file accordingly
• first, enable HAProxy stats socket in your global section (which is more a management socket
nowadays)
stats socket /var/run/haproxy/socket level admin
• the stats socket follows the bind configuration, so it can listen on a TCP port and traffic can
even be ciphered (remote management)
• then (apt|yum) install socat
• finaly, run:
socat /var/run/haproxy/socket - <<< "help"

• the following parameters can be updated into HAProxy's running configuration:
• ACL content modification: add / del
• map modification: set / add / del
• server: weight / disable / enable
• table: clear / set
• frontend: maxconn global
• global: maxconn / rate-limit

• Examples:
• Drain traffic to a server (only connection with persistence to the server will be allowed to
reach it):
socat /var/run/haproxy/socket_w - <<< "set weight b_web/s 0"
• Disable a server (set it up for maintenance):
socat /var/run/haproxy/socket_w - <<< "disable server b_web/s"
• Change a frontend's maxconn value:
socat /var/run/haproxy/socket_w - <<< "set maxconn frontend f_web 4000"

HAProxy reports
HAProxy embeds three main reporting features:
1. statistic page
In a backend or a listen section:
stats enable
stats uri /haproxy-stats
2. logs
global
log 127.0.0.1:514 local1
log 127.0.0.1:514 local2 notice
defaults
log global
option httplog # or tcplog or log-format
3. Management socket
global
stats socket /var/run/haproxy/socket

HAProxy reports - Statistic page
Useful to get a good overview of current health of the architecture.
• this page is generated on the fly and is not stored on the fileystem
• traffic statistic, at the time the statistic page was generated
• session rates during the last second
• currently establised sessions
• errors / Warnings (retries and redispatches)
• denied sessions (requests or responses)
• frontend/backend/server health (current and past)
• frontend status (OPEN or FULL (maxconn reached))
• backend status
• server status and history
• time since last status change / number of changes
• time elapsed in DOWN status
• time since last processed session

HAProxy reports - Statistic page
Example of a statistic page generated by HAProxy:

HAProxy reports - Error logs
• Error logs
Provides an accurate message when an error occurred during the connection phase.
Dec 3 18:27:14 localhost haproxy[6103]: 127.0.0.1:56059 [03/Dec/2012:17:35:10.380]
frt/f1: Connection error during SSL handshake

HAProxy reports - TCP logs
• TCP log
Information provided for a frontend / backend in TCP mode
Aug 15 19:25:13 localhost haproxy[12002]: 192.168.10.11:42991 [15/Aug/2014:19:25:13.960]
f_my b_my/m 1/0/5 58 -- 0/0/0/0/0 0/0
• It provides the following information:
• client ip and port
• date when the session started (milisecond)
• path followed by the session inside HAProxy: frontend f_my, backend b_my, server m
• timers: time spent in queues, connection time to the server, total session duration
• termination state: tells why, when and source of a session error
• number of sessions on HAProxy, the frontend, backend and server in use and the number
of retries
• number of session in queue before this one could be processed

HAProxy reports - HTTP logs
• HTTP log
Information provided for a frontend / backend in HTTP mode
Aug 15 19:15:49 localhost haproxy[10206]: 127.0.0.1:3168 [15/Aug/2014:19:15:49.413]
f_web b_web/s 4/0/0/1/5 200 11986 - - ---- 1/1/0/1/0 0/0 "GET / HTTP/1.1"
• It provides the same information as TCP plus the following ones:
• timers: time for the client to send the whole request, server process time
• HTTP status code, bytes from server to client
• captured cookies (if any)
• persistence flags
• URL (of course!!!!)

HAProxy reports - Custom logs
• Custom logs
• It is possible to declare your own log format using the log-format directive.
• Defined as a string in which we can integrate variables
• it is also possible to fetch sample from the request or response
• In example, option httplog equivalent with the log-format directive:
log-format %ci:%cp [%t] %ft %b/%s %Tq/%Tw/%Tc/%Tr/%Tt %ST %B %CC %CS %tsc
%ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs
%{+Q}r
• Same as above, but also constructs the whole URL, including the Host header field:
capture request header Host len 32
log-format %ci:%cp [%t] %ft %b/%s %Tq/%Tw/%Tc/%Tr/%Tt %ST %B %CC %CS %tsc
%ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs
"%[capture.req.method] %[capture.req.hdr(0)]%[capture.req.uri] %[capture.req.ver]"

HAProxy reports - logs: temination state
• First character: code reporting the event which caused the end of the session. Main ones are:
• -: no problem
• C: session aborted by the client
• S: session aborted by the server
• c: client side timeout expired
• s: server side timeout expired
• P: session prematurely aborted by HAProxy
• L: session locally processed by HAProxy
• Second character: TCP or HTTP session state when closing occurred. Main ones are:
• -: session terminated normally
• R: (HTTP mode only) HAProxy was waiting for a complete valid HTTP request from the client
• C: HAProxy was waiting for a connection to be established on the server
• H: (HTTP mode only) HAProxy was waiting for a complete valid response headers from the
server
• D: the session was in the DATA phase

HAProxy reports - logs: temination state
examples
• Combination of termination states can be used to trigger some cases like:
• client or server side network outage
• application or server slowness or issues
• Most common termination states are:
• CD: the client aborted the session when HAProxy was sending it the response
• cD: HAProxy's client timeout triggered because not data has been acknowledged by the client
(device turned off, network outage, etc...)
• SD: the server aborted the session while sending the response. In HTTP mode, a 502 is
generated
• sD: HAProxy's timeout server triggered because either the server was too slow to generate a
response (HTTP mode only, HAProxy returns a 504) or too slow to acknowledge data.

Weakness in MySQL client library
The information below is independent from HAProxy. It is purely related to MySQL.
• MySQL clients don't close connections properly:
• They send the layer 7 "QUIT" and shut the connection without waiting for the server to confirm
the sequence
• the connection is shut a TCP FIN
• This is due to the way the MySQL client library manages the close
Mysql Client ==> "QUIT" sequence ==> Mysql Server
Mysql Client ==> FIN ==> MySQL Server
Mysql Client <== FIN ACK <== MySQL Server
Mysql Client ==> ACK ==> MySQL Server
• A TCP FIN means the system must wait for the tcp TIME_WAIT before re-using the source port
to establish a new connection to the couple [dest IP]:[dest port]
• This becomes a problem when the connection rate is high (several hundreds per second),
since it can lead to a tcp source port exhaustion.
• by default, TIME_WAIT is 2 times MSL (Maximum Segment Lifetime), so 240s, but force at
60s in Linux kernel
• TIME_WAIT main purpose is to prevent potential overlap with new connections

Weakness in MySQL client library
• The source ports exhaustion leads HAProxy to generate the no free ports error log
message
• Rule to compute the maximum connection rate to a MySQL server:
Max conn/s = (ip_local_port_range[max] - ip_local_port_range[min]) / TIME_WAIT
• For default values: Max conn/s = (61000 - 32768) / 60 = 470
• With recommended values: Max conn/s = (65024 - 1024) / 60 = 1066
• It is recommended to let HAProxy to manage TCP source ports. It is much more efficient than
the kernel to do it.
• Use multiple IPs configured on HAProxy server to increase the Max conn/s rate and let
HAProxy to manage available TCP source port:
server mysql1 10.0.0.1:3306 check source 10.0.0.100:1024-65024
server mysql1_bis 10.0.0.1:3306 check source 10.0.0.101:1024-65024

HAProxy and MySql overview
• HAProxy just forward TCP connections from MySQL client to MySQL server
• HAProxy is not aware of request nor response content: MySQL is only treated as payload
• MySQL clients have two main ways of working:
• short live on-demand connections established, used then closed (PHP way)
• persistent connections established and maintained by the client and used as a pool of
resources (the JAVA way)
• more information from severalnines.com:
http://www.severalnines.com/resources/clustercontrol-mysql-haproxy-load-balancing-tutorial

Hints for short live connections - Sysctls
sysctls to take care of for performance purpose:
• net.ipv4.ip_local_port_range: local port range allowed to reach a server IP address.
The wider, the best.
default: "32768 61000", recommended: "1024 65024"
• net.ipv4.tcp_max_syn_backlog: required to support high connection rate (and better
support SYN floods attacks)
default: 1024, recommended: 60000
• net.ipv4.tcp_tw_reuse: Allow early reuse of a same source port for outgoing connections.
• net.core.somaxconn: per socket outstanding connect requests that have not been
accepted/serviced yet by HAProxy

Hints for short live connections - CPU
• Remember HAProxy's design: single process, single thread, 100% event driven => this means
HAProxy performs very well with a single CPU core !!!!
We can easily reach 50K HTTP connections per second with a single CPU core!
• CPU is very important. Prefer the frequency over the number of cores. The bigger the CPU
cache, the best too
• Whenever possible, bind network interrupts to a CPU core and HAProxy to one close to it
• Use cpu-map HAProxy's global directive to let it bind itself to the core you want
# process-id cpu-id
cpu-map 1 1
• DISABLE and uninstall IRQ Balance
• avoid VMs when high connections rate is required (high means >5K/s)

Hints for short live connections - HAProxy
• What's are the impact of reloading HAProxy's configuration?
Since connections last a very short time (less than a second), all the clients will switch to the
new process quickly, so no impact when reloading the configuration.
• What are the impact of running HAProxy in multiprocess mode?
No. Since server's maxconn (queueing management) and health checks will be performed
per process
No. Since it is impossible to synchronise stick tables (some people use stick table to store the
MySQL master server id)

Hints for persistent connections - Sysctl
sysctls to take care of for performance purpose:
• net.ipv4.ip_local_port_range: local port range allowed to reach a server IP address.
The wider, the best.
default: "32768 61000", recommended: "1024 65534"
• net.ipv4.tcp_rmem: minimum, default and maximum read buffer size (in bytes) allocated per
socket
• net.ipv4.tcp_wmem: minimum, default and maximum write buffer size (in bytes) allocated per
socket

Hints for persistent connections - HAProxy
• What are the impact of reloading HAProxy's configuration?
Since connections are established for a very long time, they'll be managed by the old process
No connections will be killed. That said, it is currently not possible to monitor neither manage
connections established on the old process
• What are the impact of running HAProxy in multiprocess mode?
server's maxconn (queueing management) and health checks will be performed per process
it is impossible to synchronise stick tables (some people use stick table to store the MySQL
master server id)
• Don't use HAProxy's queueing mechanism since your connections are established for a long
time, new ones may be queued for a long time as well.

Hints for persistent connections - HAProxy
• Don't hesitate to calculate HAProxy's memory footprint required for the number of connections
you need. For each MySQL connection passing through HAProxy:
• 1 read and 1 write kernel buffer for both client and server connection (sysctls tcp_rmem
and tcp_wmem)
•2 HAProxy buffers (tune.bufsize)
memory footprint = 2 * tcp_rmem[min] + 2 * tcp_wmem[min] + 2 * tune.bufsize
= 2 * 8K + 2 * 8K + 2 * 16K
= 48KBytes
If HAProxy has to maintain 1000 connections to a MySQL cluster, it needs around 50 MBytes
of memory
• The rule above is a bit simplified, add 10% of margin
• if SSL offloading is enable, also add 2 SSL buffers of 64KB each...

Active/active failover scenarios
• You are responsible to failover the traffic from one HAProxy to the other one
• Different ways exist to ensure high availability of services hosted by HAProxy
• VRRP: use keepalived crossed VIPs
• DNS: use at your own risks
• RHI: use a routing protocols (iBGP, OSPF) to anounce availability of the server (bird,
exabgp)
• The choice of high availability scenario is driven by:
• the main purpose to achieve
• network architecture topology and limitations
• network latency between locations
• compatibility of the application being load-balanced
• Whatever the chosen scenario, HAProxy configuration can help:
• use local server first
• if local servers are unavailable, then forward the traffic to the other datacenter, if available
• generates HTTP redirects

Active/active failover scenarios - VRRP
• Active/active scenario with VRRP:
• if one HAProxy node fails, both VIPs will be hosted by the remaining one

• requires a layer 2 link for VRRP to work, not compatible with clouds
• VLANs must be shared by both datacneter
• can hardly scale over 2 DC
• failover takes up to 3s

Active/active failover scenarios - DNS
• Active/active scenario with DNS:

• if one HAProxy node fails, DNS must be updated accordingly and quickly
• HAProxy's configuration can help:
• provide a URL to be monitored by DNS servers
• can route requests to the remaining node during DNS propagation
• no links between the datacenter or cloud provider is required
• scaling is linked to application capabilities and to sensitivity to latency
• failover is unpredictible. 90% of users fail over quickly, no idea for the last 10%
• some ISPs rewrite DNS response TTL to a long value (20m for mine)
• IP1 and IP2 could be hosted by VRRP in each datacenter

Active/active failover scenarios
• Confguration example to detect application availability per datacenter and failover
• the IP address (or DNS resolution) should also failover to the remaining datacenter
frontend ft_dc1
bind 10.0.0.1:3306
acl bk_dc1_DOWN nb_srv(bk_dc1) eq 0
acl bk_dc2_UP nb_srv(bk_dc2) ge 1
# fail over to DC2 if:
# - no more servers in bk_dc1
# - still some servers in bk_dc2
use_backend bk_dc2 if bk_dc1_DOWN bk_dc2_UP
# default rule
default_backend bk_dc1
frontend ft_dc2
bind 10.0.0.2:3306
acl bk_dc2_DOWN nb_srv(bk_dc2) eq 0
acl bk_dc1_UP nb_srv(bk_dc1) ge 1
# fail over to DC1 if:
# - no more servers in bk_dc2
# - still some servers in bk_dc1
use_backend bk_dc1 if bk_dc2_DOWN bk_dc1_UP
# default rule
default_backend bk_dc2
backend bk_dc1
server mysql1 10.0.0.11:3306 name mysql maxconn 1000 check
backend bk_dc2
server mysql2 10.0.0.12:3306 name mysql maxconn 1000 check

Security considerations
• filter access to your MySQL frontends:
frontend ft_dc1
bind 10.0.0.1:3306
tcp-request content reject unless { src 10.0.0.0/24 }
• limit the number of active connections per source IP:
# Table definition
stick-table type ip size 100k expire 30s store conn_cur
tcp-request connection track-sc1 src
# Allow clean known IPs to bypass the filter
tcp-request connection accept if { src -f /etc/haproxy/whitelist.lst }
# Shut the new connection as long as the client has already 10 opened
tcp-request connection reject if { sc1_conn_cur ge 10 }
• slowloris protection: (HTTP)
timeout http-request 10s

Performance Tuning of HAProxy for Database Load Balancing - Slides

Recommended

Recommended

More Related Content

More from Severalnines

More from Severalnines (20)

Recently uploaded

Recently uploaded (20)

Performance Tuning of HAProxy for Database Load Balancing - Slides