"What would you be if you were attached to   another object by an inclined plane, wrapped   helically around an axis?"
"Screwed."
"There you go."

The Big Bang Theory Season 4 Episode 5

HAProxy

An introduction and how openSUSE is using it

Created by Marcus Rückert or just darix ;)

About me

• S.u.S.E. user since 5.1
• With SuSE since 2005
• Doing openSUSE and SUSE infrastructure nowadays
• Love to try out new toys → some even end up in the distro or in our infrastructure

What is it?

• The Reliable, High Performance TCP/HTTP Load Balancer
  • Also SSL endpoint now (1.5 dev branch)
• Opensource
  • You can get commercial support/appliance
• Runs many high traffic sites
  • discqus, reddit, farmville, fedora, instagram, imgur, w3.org ...
• http://haproxy.1wt.eu/

What does it do?

• reverse proxy
• load balancer
• use cases
  • scale out
  • fail over

Numbers matter

• can actually saturate 10gig links with modest hardware
  • 20% cpu usage
  • 100 000 sessions/s
  • http://haproxy.1wt.eu/10g.html
• 572000 reqs/s
• often the benchmark tools are their at limits
• a bit slower with ssl requests, but thats in openssl
• supports TCP Fastopen, splice feature on linux

We want it secure

• since 1.5-dev12
• supports all the nice features of openssl
  • SNI
  • NPN
  • yes also Heartbleed ;)

what about ssl on 1.4

• you can still load balance in tcp mode e.g. to stud/stunnel
• Even with load balancing based on ssl sessions

Other cool bits

• seamless restart (was broken with systemd, should be fixed now)
• powerful matchers and request routing tools (session stickiness)
• load balancing with server limit taken into account
• health checks for various protocols
• nice tools around it:
  • hatop
  • controlling your running instance via the control socket

Installing

haproxy-1.5 package in server:http

Warning: All examples are tested with 1.5 only. They might work with 1.4 though.

Diving in

Simple Config

Global section

global
  maxconn 32768
  chroot /var/lib/haproxy
  user haproxy
  group haproxy
  daemon
          

Simple Config

Defaults

defaults
  log     global
  mode    http
  option  httplog
  option  dontlognull
  option  redispatch
  retries 3
  maxconn 10000
  timeout connect     5000
  timeout client     50000
  timeout server    450000
          

apply to all blocks following unless overwritten.

Simple Config

Actually serving something

listen stats
  bind :80
  stats enable
  stats uri     /
  stats refresh 5s
          

First start

http://localhost/

Detailed Statistics

Hover over values with dotted underline to get more detailed stats

Doing some real work

HTTP for starters

listen proxy-http
  bind :8080
  server stats01 127.0.0.1:80
          

http://localhost:8080/

Doing some real work

TCP mode

listen proxy-tcp
  bind :8081
  mode tcp
  option tcplog
  server stats01 127.0.0.1:80
          

http://localhost:8081/

Splitting frontend and backend

frontend fe-stats
  bind :8082
  default_backend stats

backend be-stats
  stats enable
  stats uri     /
  stats refresh 5s
          

http://localhost:8082/

Routing requests

frontend fe-routing
  bind :8083
  acl is_localhost hdr(host) -i localhost
  use_backend be-stats if is_localhost
  default_backend be-lighty

backend be-lighty
  option forwardfor
  option httpchk HEAD / HTTP/1.1\r\nHost:\ localhost
  server lighty2 127.0.0.1:81 check
  server lighty1 127.0.0.1:82 check
          

Gimme SSL

Add this to the initial listen block. The bind statement should be one line without the backslashes.

haproxy.pem contains key, cert, chain certs, dh parameters.

bind :443 tfo ssl crt \
  /etc/ssl/private/haproxy.pem no-sslv3 \
  npn http/1.1,http/1.0 \
  ciphers HIGH:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK:!RC4:!ADH:!LOW@STRENGTH

acl is_ssl ssl_fc
acl is_stats hdr(host) -i stats.example.com
redirect code 301 prefix https://stats.example.com if is_stats !is_ssl
          

More on SSL - tell the backend

reqidel ^X-Forwarded-(Proto|Ssl).*
reqidel ^HTTPS.*

reqadd HTTPS:\ on  if is_ssl
reqadd X-Forwarded-Ssl:\ on if is_ssl

reqadd X-Forwarded-Proto:\ https if is_ssl
reqadd X-Forwarded-Protocol:\ https if is_ssl

reqadd X-Forwarded-Proto:\ http unless is_ssl
reqadd X-Forwarded-Protocol:\ http unless is_ssl

rspadd Strict-Transport-Security:\ max-age=31536000 if is_ssl
          

MySQL anyone?

listen mysql
  bind :3306
  timeout client 5m
  timeout connect 10s
  timeout server 5m
  mode tcp
  option tcplog
  option mysql-check user haproxy_check
  server mysql1 192.168.0.1:3306 check backup
  server mysql2 192.168.0.2:3306 check
          

Logging

syslog based

Global section

log 127.0.0.1 daemon info
log-send-hostname <hostname>-haproxy
           

Defaults section

option  log-health-checks
option  log-separate-errors
option  dontlog-normal
          

This example will only log error cases but not normal requests.

@openSUSE

Where?

virtualization cluster for openSUSE infrastructure

Goals

• minimize the surface to the outside
• minimize hosts that need SSL cert+key
• freebie: all hosts on the cluster get SSL without much trouble

Which services?

HTTP(S), git, MySQL

For ssh we are using xinetd

Actual implementation

• VM itself.
• 1 frontend block per "physical" interface
• handles http and https
• tcp frontend for GIT
• tcp frontend for MySQL
• mapping external hostnames to VMs
• mapping external services for the VMs (download.o.o)
• services on the VMs are only accessible by the proxy but not other VMs

Q & A

Thank you for flying with openSUSE