View source for Linux Cluster with HAProxy and Keepalived

= HAproxy guide =
The goal of the team behind HAProxy is to provide a “free, very fast and reliable solution” for load-balancing TCP and HTTP-based applications. Because of this HAProxy is considered by many to be the de facto standard when it comes to software-based load balancing and is
currently being used by sites such as GitHub, Reddit, Twitter and Tumblr to name a few.

It has been designed to run on Linux, Solaris, FreeBSD, OpenBSD as well as AIX platforms. While it’s designed to run on most x86-64 hardware that has limited resources, it will perform best when provided enterprise-grade hardware such as 10+ Gig NIC’s and Xeon class CPU’s
or similar.
== Terminology ==
Below are a few of the key terms and concepts you should understand when working with HAProxy. When working with load balancers, these are the key concepts that will apply to all solutions. 
=== Frontend ===
Frontend, within the context of HAProxy, dictates where and how incoming traffic is routed to machines behind HAProxy. They allow you to setup rules (ACL’s) that will “watch” for specific URL syntax inbound and outbound of the load balancer and intelligently route a user’s traffic as needed.

Furthermore, the frontend is where you will configure what IP and ports that HAProxy will listen for traffic on as well as configuring HTTPS on those respective ports. 

Example frontend:
<source>
frontend http-in
    bind *:80
    default_backend appX-backend 
</source>
This frontend example shows that we are only opening port 80 for incoming requests and redirecting all traffic to the default backend called appX-backend. 
=== Backend === 
As the name suggests, a backend within the context of HAProxy is a group of resources that is home to your data or applications. These backend resources are where traffic will get routed by the rules you have configured in your frontends. 

Backend resource groups can consist of a single server or multiple servers but in the context of load balancers, it is assumed that a minimum of two are being used. The more backend resources you can add to a group; the lower the load on each individual resource will be while increasing the number of users you can serve at any given time. 

Lastly, just as with the frontend, you will also configure the IP and ports that your backend resources are listening for requests on. 

Example backend:
<source>
backend appX-backend
    balance roundrobin
    server appX_01 192.168.2.2:8080 check
    server appX_02 192.168.2.3:8080 check
</source>
In this example, the backend is called appX-backend and it contains two servers that are being accessed using roundrobin which is explained later in this guide. 

=== ACL (Access Control List) ===
In the context of HAProxy, ACLs are the backbone of more in-depth and complex configurations that contain multiple frontends as well as multiple backends and need very precise routing.

With ACLs, you have the ability to parse through requests and do a multitude of different actions such as rewriting and redirecting all traffic requests as needed. Because HAProxy has the ability to load balance over Layer 4 or Layer 7 in the OSI model, you can effectively configure it to handle a number of different uses at the same time with multiple frontends and backend.

Example ACL:
<source>
frontend http-in
    bind *:80
    acl url_appX path_beg -i /appX/
    use_backend appX-backend if url_appX
    default_backend appZ-backend 
</source>
This example is rather simple in that it evaluates the incoming request for the resource path which is the context immediately after the first / such as http://example.com/appX. In this scenario, if the incoming request is http://example.com/appX, the request is sent to the backend called appX-backend whereas all other requests will default to the appZ-backend.
=== Algorithms ===
HAProxy comes with a fairly large number of options when it comes to choosing the method in which you want requests to be served to your backend resources. Below are a few of the more common ones as well as a short description of how each will work. 
=== Round Robin ===
Each server is used in a never-ending line, starting with the first one listed in a given backend until the end of that list is reached at which point the next request will go back to the first resource again. By default, HAProxy will use this algorithm if one is not specified when building a backend. 
=== Least Connection ===
Each resource in a given backend is evaluated to determine which one has the least number of active connections. The resource with the lowest number will receive the next request. The developers of HAProxy state that this algorithm is a great option for connections that are expected to last a long time such as LDAP and SQL but not for HTTP.       
== Install ==
Guide created on CentOS 7.6 (Core)
<source lang=bash>
sudo yum install -y  haproxy
sudo systemctl enable haproxy.service

# check firewall
sudo firewall-cmd --list-all
[sudo] password for cloud_user:
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: eth0
  sources:
  services: dhcpv6-client http ssh # <- note http:80
  ports:                           # is allowed
  protocols:
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:

# add permanent ingress rule to allow port 8080
sudo firewall-cmd --permanent --add-port=8080/tcp
sudo firewall-cmd --reload
...
  ports: 8080/tcp
...
</source>

== Structure ==
=== Files & Folders === 
Below are a number of the directories you can expect to find in a default HAProxy installation on Debian-based OS but regardless of the host OS, the files will be essentially the same with relation to what they do. While there are a number of files and directories listed, the one that will matter the most is the haproxy.cfg file located at /etc/haproxy/haproxy.cfg.

# /usr/sbin/haproxy - Default location of the binary.
# /usr/share/doc/haproxy - Built-in documentation.
# /etc/init.d/haproxy - init script used to control the HAProxy process/service.
# /etc/default/haproxy - File that is sourced by both the initscript for haproxy.cfg file location.
# /etc/haproxy - Default location of the haproxy.cfg file which determines all functions of HAProxy.

=== Configuration File ===
HAProxy functions are completely controlled by the haproxy.cfg file. This file is where you will build your frontends and backends as well as other various settings which are described below. For example, here is an example of a basic haproxy.cfg file.
<source>
global
    log 127.0.0.1 local0
    daemon
    maxconn 256
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode http
    timeout connect 5000ms
    timeout client 50000ms
    timeout server 50000ms
    option forwardfor
    option http-server-close
#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend http-in
    bind *:80
    rspadd X-Forwarded-Host:\ http:\\\\example.com
    default_backend appZ-backend

backend appZ-backend
    balance roundrobin
    server appZ_01 IP_OF_MACHINE_01:8080 check
    server appZ_02 IP_OF_MACHINE_02:8080 check
    server appZ_03 IP_OF_MACHINE_03:8080 check
    server appZ_04 IP_OF_MACHINE_04:8080 check 
</source>
With this example, you can see how the different pieces discussed are brought together to create a working configuration. While the introduction of the global and default fields is new, the syntax should be similar enough to the frontend and backend that you are able to determine what their functions are. With that said, below are a few explanations for a few of these new lines that may not seem immediately obvious if you have never seen them before. 
mode http

As mentioned previously, HAProxy has the ability to load balance using layer 4 or 7 in the OSI model. With that in mind and by setting the mode to http, HAProxy can now inspect HTTP headers for all requests and modify and redirect per each request.  The other option is to use tcp as the mode which allows HAProxy to route per IP and port; ignoring all HTTP headers. 

Reference: https://cbonte.github.io/haproxy-dconv/1.6/configuration.html#mode

=== timeout 's ===
HAProxy has the ability to specify timeouts for various aspects of standard HTTP/TCP protocols. In the example, the configuration file has been configured to limit connection (time to wait for a successful connection attempt), client (maximum client side inactivity time) and server (maximum server side inactivity time) time frames. 

By setting various timeout options, HAProxy can efficiently manage the amount of resources it needs to continually consume which can be freed for new connections. Because there are a large number of timeouts available, referencing the documentation will provide a much more thorough overview of what is possible. 

Reference: https://cbonte.github.io/haproxy-dconv/1.6/configuration.html#timeout 

=== option forwardfor ===
Enables the use of X-Forwarded-For headers by HAProxy.
=== option http-server-close ===
This option allows you to configure how HAProxy handles connections from the server side. HAProxy by default runs in keep-alive mode which means that connections are kept open and in an idle state. By using this option, we can force HAProxy to close these connections so that they are not consuming resources.

Reference: https://cbonte.github.io/haproxy-dconv/1.6/configuration.html#option%20http-server-close
=== rspadd X-Forwarded-Host:\===
As the forwardfor option enables the use of the X-Forwarded-For header, it must now be configured and that is what rspadd is used for. One of the more common uses of this is when hosting various applications behind HAProxy and you want to ensure that your end users only see responses coming from HAProxy and not the machines behind it which can occasionally occur with Asynchronous processes.

== Usage ==
To bring the above information into further context, below are a few basic example configurations to help illustrate how one would setup HAProxy for different scenarios. 
=== Active/Passive ===
This configuration can be used when you need to route all traffic to one machine until it goes offline - at which time, HAProxy will automatically begin to route traffic to the second machine. 
<source>
backend appZ-backend
    server appZ_01 192.168.2.2:8080 check 
    server appZ_02 192.168.2.3:8080 check backup 
</source>
=== ACL Request ===
With this configuration, you can route a given incoming request to different backends as needed. If you run multiple applications behind HAProxy, this will allow you to add additional backend resources as needed to each individual application instead of increasing both
at the same time. 
<source>
frontend http-in
    bind *:80
    rspadd X-Forwarded-Host:\ http:\\\\example.com
    acl url_appY path_beg -i /appY/
    use_backend appY-backend if url_appY
    default_backend appZ-backend
    
backend appY-backend
    balance roundrobin
    server appY_01 192.168.2.2:8080 check 
    server appY_02 192.168.2.3:8080 check
 
backend appZ-backend
    server appZ_01 192.168.2.4:8080 check
    server appZ_02 192.168.2.5:8080 check backup 
</source>
== [https://linuxacademy.com/guide/14569-configuration-of-haproxy/ Configuration of HAProxy] ==

= Application Stack With HAproxy, Keepalived and postfix  example =

Outgoing Mail is directed via a dns record dc1server00.home.local or dc2server00.home.local dependant on the datacenter to the vrrp address that is maintained by keepalived. 

Haproxy is listening on port 25 (smtp) on both nodes in an active/standby fashion and is configured to direct connections to postfix on both nodes (Active/Active), therefore both nodes in the cluster are live and servicing emails at all times. 

Postfix will then via normal MX delivery, deliver the email to the customer.

[[File:Keepalived.png|none|left|Keepalived and HAproxy example system]]

== Keepalived ==
Keepalived uses the vrrp(2) protocol to maintain a virtual ip address that is able to float between any node in a cluster, for this system there is a primary and backup server set by priority in the configuration.
keepalived is configured to monitor that haproxy is running on the master, if it isn’t running then the priority of the master will be reduced and the backup which will now be of higher priority will takeover the floating address. 

Setup, on both servers in the cluster:

Install keepalived via the yum unix command (as root)
 #yum install keepalived

add the following  line to the bottom of <code>/etc/sysctl.conf</code>. This allows processes to bind to sockets on the machine even if the ip does not exist. 
 net.ipv4.ip_nonlocal_bind = 1

 # sysctl -p

Enable keepalived to be started up on boot.
 # systemctl enable keepalived.service

{| class="wikitable"
|+ keepalived config
|- 
! DC1
! DC2
|- style="vertical-align:top;"
| <source lang=bash>cat /etc/keepalived/keepalived.conf
# GLOBAL SETTINGS
global_defs {
   notification_email {
       support@example.com #Set from Ansible haproxy deployment
   }
  notification_email_from keepalive@dc1server.home.local 
   smtp_server 172.31.101.115 # ip address for smtp_relay
   smtp_connect_timeout 20
}
vrrp_script chk_haproxy {
   script "systemctl status haproxy" # verify the pid existence
   interval 2		    # check every 2 seconds
   weight 2		    # add 2 points of prio if OK

}
vrrp_instance dc1server
    state MASTER
    interface eth0
    virtual_router_id 50
    priority 101
    advert_int 1
    track_interface {
      eth0
    }
    track_script {
      chk_haproxy weight 20
    }
    authentication {
      auth_type PASS
      auth_pass SOMEPASSWORD! 
    }
    virtual_ipaddress {
      10.0.10.3 dev eth0 
    }
}</source>
| <source lang=bash>cat /etc/keepalived/keepalived.conf
# GLOBAL SETTINGS
global_defs {
  notification_email {
    support@example.com # set from Ansible haproxy deployment
   }
  notification_email_from keepalive@dc2server.home.local 
    smtp_server 172.31.101.115 # ip address for robot
    smtp_connect_timeout 20
}
vrrp_script chk_haproxy {
    script "systemctl status haproxy" # verify the pid existence
    interval 2		    # check every 2 seconds
    weight 2		    # add 2 points of priority if OK

}
vrrp_instance dc2server
    state MASTER
    interface eth0
    virtual_router_id 50
    priority 101
    advert_int 1
    track_interface {
      eth0
    }
    track_script {
      chk_haproxy weight 20
    }
    authentication {
      auth_type PASS
      auth_pass SOMEPASS 
    }
    virtual_ipaddress {
      10.0.20.3 dev eth0
    }
}
</source>
|}

== HAproxy ==
The function of HAproxy is to load balance connections to postfix on either node in the cluster, This allows for an “Active/Active” setup. Because there is no way to tell the workload of what has been submitted to each node I have opted for a simple Round Robin load balancer algorithm. it will invoke an smtp HELO to check if the postfix ports are available and listening. Haproxy will be configured to present statistics and the ability to drain servers via a browser at http://hostname:9000/ .

;Setup on both servers in the cluster

install haproxy via the yum command
 # yum install haproxy

enable haproxy to be started up on boot.
 # systemctl enable haproxy.service


{| class="wikitable"
|+ haproxy config
|- 
! DC1
! DC2
|- style="vertical-align:top;"
| <source lang=bash style="white-space: pre-wrap;">cat /etc/haproxy/haproxy.cfg
global
    chroot /var/lib/haproxy
    pidfile /var/run/haproxy.pid
    maxconn 4000
    user haproxy
    group haproxy
    daemon
    stats socket /var/lib/haproxy/stats
    log 172.31.0.254 local2
    ssl-default-bind-ciphers  kEECDH+aRSA+AES:kRSA+AES:+AES256:RC4-SHA

defaults
    mode tcp
    log global
    option tcplog
    option dontlognull
    option redispatch
    retries 3
    timeout queue 1m
    timeout connect 10s
    timeout client 1m
    timeout server 1m
    timeout check 10s
    maxconn 3000

listen stats
    bind *:9000 ssl crt /etc/pki/tls/private/
    mode http   
    stats enable
    stats hide-version
    stats uri /
    stats realm HAProxy\ Statistics
    stats auth admin:p@ssw0rd
    stats admin if TRUE

frontend smtpdc1 
    bind 10.0.10.3:25
    use_backend emailservice

backend emailservice
    description Emailservice
    balance roundrobin
    option tcplog
    option smtpchk HELO haproxy
    server dc1server01 10.0.10.1:25 check 
    server dc1server02 10.0.10.2:25 check </source>
| <source lang=bash>cat /etc/haproxy/haproxy.cfg
global
    chroot /var/lib/haproxy
    pidfile /var/run/haproxy.pid
    maxconn 4000
    user haproxy
    group haproxy
    daemon
    stats socket /var/lib/haproxy/stats
    log 172.31.0.254 local2
    ssl-default-bind-ciphers  kEECDH+aRSA+AES:kRSA+AES:+AES256:RC4-SHA

defaults
    mode tcp
    log global
    option tcplog
    option dontlognull
    option redispatch
    retries 3
    timeout queue 1m
    timeout connect 10s
    timeout client 1m
    timeout server 1m
    timeout check 10s
    maxconn 3000

listen stats
    bind *:9000 ssl crt /etc/pki/tls/private/
    mode http   
    stats enable
    stats hide-version
    stats uri /
    stats realm HAProxy\ Statistics
    stats auth admin:p@ssw0rd
    stats admin if TRUE

frontend smtpdc1 
    bind 10.0.20.3:25
    use_backend emailservice

backend emailservice
    description Emailservice
    balance roundrobin
    option tcplog
    option smtpchk HELO haproxy
    server dc2server01 10.0.20.1:25 check 
    server dc2server02 10.0.20.2:25 check
</source>
|}

== Troubleshooting ==
 [root@dc1server01 ~]# fgrep -i keep /var/log/messages* 
 /var/log/messages-20160822:Aug 22 09:30:58 dc1server01 Keepalived[1141]: Starting Keepalived v1.2.13 (11/18,2014)
 /var/log/messages-20160822:Aug 22 09:30:58 dc1server01 Keepalived[1563]: Starting Healthcheck child process, pid=1564
 /var/log/messages-20160822:Aug 22 09:30:58 dc1server01 Keepalived[1563]: Starting VRRP child process, pid=1565
 /var/log/messages-20160822:Aug 22 09:30:58 dc1server01 Keepalived_vrrp[1565]: Netlink reflector reports IP 10.0.10.1 added
 /var/log/messages-20160822:Aug 22 09:30:58 dc1server01 Keepalived_vrrp[1565]: Netlink reflector reports IP fe80::215:5dff:fee3:5004 added
 /var/log/messages-20160822:Aug 22 09:30:58 dc1server01 Keepalived_vrrp[1565]: Registering Kernel netlink reflector
 /var/log/messages-20160822:Aug 22 09:30:58 dc1server01 Keepalived_vrrp[1565]: Registering Kernel netlink command channel
 /var/log/messages-20160822:Aug 22 09:30:58 dc1server01 Keepalived_vrrp[1565]: Registering gratuitous ARP shared channel
 /var/log/messages-20160822:Aug 22 09:30:58 dc1server01 Keepalived_vrrp[1565]: Opening file '/etc/keepalived/keepalived.conf'.
 /var/log/messages-20160822:Aug 22 09:30:58 dc1server01 Keepalived_vrrp[1565]: Truncating auth_pass to 8 characters
 /var/log/messages-20160822:Aug 22 09:30:58 dc1server01 Keepalived_vrrp[1565]: Configuration is using : 66367 Bytes
 /var/log/messages-20160822:Aug 22 09:30:58 dc1server01 Keepalived_vrrp[1565]: Using LinkWatch kernel netlink reflector...
 /var/log/messages-20160822:Aug 22 09:30:58 dc1server01 Keepalived_vrrp[1565]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
 /var/log/messages-20160822:Aug 22 09:30:58 dc1server01 Keepalived_vrrp[1565]: VRRP_Script(chk_haproxy) succeeded
 /var/log/messages-20160822:Aug 22 09:30:58 dc1server01 Keepalived_healthcheckers[1564]: Netlink reflector reports IP 10.0.10.1 added
 /var/log/messages-20160822:Aug 22 09:30:58 dc1server01 Keepalived_healthcheckers[1564]: Netlink reflector reports IP fe80::215:5dff:fee3:5004 added
 /var/log/messages-20160822:Aug 22 09:30:58 dc1server01 Keepalived_healthcheckers[1564]: Registering Kernel netlink reflector
 /var/log/messages-20160822:Aug 22 09:30:58 dc1server01 Keepalived_healthcheckers[1564]: Registering Kernel netlink command channel
 /var/log/messages-20160822:Aug 22 09:30:58 dc1server01 Keepalived_healthcheckers[1564]: Opening file '/etc/keepalived/keepalived.conf'.
 /var/log/messages-20160822:Aug 22 09:30:58 dc1server01 Keepalived_healthcheckers[1564]: Configuration is using : 7586 Bytes
 /var/log/messages-20160822:Aug 22 09:30:58 dc1server01 Keepalived_healthcheckers[1564]: Using LinkWatch kernel netlink reflector...
 /var/log/messages-20160822:Aug 22 09:30:58 dc1server01 Keepalived_vrrp[1565]: VRRP_Instance(dc1server) Transition to MASTER STATE
 /var/log/messages-20160822:Aug 22 09:30:58 dc1server01 Keepalived_vrrp[1565]: VRRP_Instance(dc1server) Received higher prio advert
 /var/log/messages-20160822:Aug 22 09:30:58 dc1server01 Keepalived_vrrp[1565]: VRRP_Instance(dc1server) Entering BACKUP STATE
 /var/log/messages-20160822:Aug 22 09:30:58 dc1server01 Keepalived_healthcheckers[1564]: Netlink reflector reports IP fe80::215:5dff:fee3:5004 added
 /var/log/messages-20160822:Aug 22 09:30:58 dc1server01 Keepalived_vrrp[1565]: Netlink reflector reports IP fe80::215:5dff:fee3:5004 added
 /var/log/messages-20160822:Aug 22 09:38:29 dc1server01 Keepalived_vrrp[1565]: VRRP_Instance(dc1server) Transition to MASTER STATE
 /var/log/messages-20160822:Aug 22 09:38:30 dc1server01 Keepalived_vrrp[1565]: VRRP_Instance(dc1server) Entering MASTER STATE
 /var/log/messages-20160822:Aug 22 09:38:30 dc1server01 Keepalived_vrrp[1565]: VRRP_Instance(dc1server) setting protocol VIPs.
 /var/log/messages-20160822:Aug 22 09:38:30 dc1server01 Keepalived_vrrp[1565]: VRRP_Instance(dc1server) Sending gratuitous ARPs on eth0 for 10.0.10.3
 /var/log/messages-20160822:Aug 22 09:38:30 dc1server01 Keepalived_healthcheckers[1564]: Netlink reflector reports IP 10.0.10.3 added
 /var/log/messages-20160822:Aug 22 09:38:35 dc1server01 Keepalived_vrrp[1565]: VRRP_Instance(dc1server) Sending gratuitous ARPs on eth0 for 10.0.10.3
 /var/log/messages-20160822:Aug 22 09:44:53 dc1server01 Keepalived_vrrp[1565]: VRRP_Instance(dc1server) Received lower prio advert, forcing new election


 [root@dc2server01 ~]# fgrep -i keep /var/log/messages*
 /var/log/messages:Aug 22 10:53:04 dc2server01 Keepalived_vrrp[1234]: VRRP_Script(chk_haproxy) failed
 /var/log/messages:Aug 22 10:53:05 dc2server01 Keepalived_vrrp[1234]: VRRP_Instance(dc2server) Received higher prio advert
 /var/log/messages:Aug 22 10:53:05 dc2server01 Keepalived_vrrp[1234]: VRRP_Instance(dc2server) Entering BACKUP STATE
 /var/log/messages:Aug 22 10:53:05 dc2server01 Keepalived_vrrp[1234]: VRRP_Instance(dc2server) removing protocol VIPs.
 /var/log/messages:Aug 22 10:53:05 dc2server01 Keepalived_healthcheckers[1233]: Netlink reflector reports IP 10.0.20.3 removed
 /var/log/messages:Aug 22 11:04:49 dc2server01 Keepalived_vrrp[1234]: VRRP_Script(chk_haproxy) succeeded


;Multiline config
You can't do multiline syntax in the haproxy.cfg [https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#2.1 defined format documentation]