how do you deal with SPOF with HAproxy
-
Hi,
I wonder if this ever came across where you use HAproxy but wish to made it redundant and can sustain it going down. I know it rarely ever go down but what do you do if you want to make it redundant.
-
@Emad-R said in how do you deal with SPOF with HAproxy:
Hi,
I wonder if this ever came across where you use HAproxy but wish to made it redundant and can sustain it going down. I know it rarely ever go down but what do you do if you want to make it redundant.
You'd be looking at changing DNS via scripts if you need to have a high availability HAproxy. Totally doable with any DNS provider that gives API access.
-
what about multiple A/AAAA records for multiple HAproxy ?
-
@Emad-R said in how do you deal with SPOF with HAproxy:
what about multiple A/AAAA records for multiple HAproxy ?
I know it's possible because Google does just that. How you'd go about it I don't know. Even if you did, you'd still have to script removing one that goes down, otherwise computers are still going to use the IP that is down.
-
@Emad-R said in how do you deal with SPOF with HAproxy:
I wonder if this ever came across where you use HAproxy but wish to made it redundant and can sustain it going down.
I think that's the main use case of it.
-
@Emad-R said in how do you deal with SPOF with HAproxy:
what about multiple A/AAAA records for multiple HAproxy ?
That handles round robin, not failover. You need real HA. This is how HA-Proxy is meant to be used...
-
DNS round robin for failover kind of works, but it's not the greatest solution. We use it for our nginx proxies.
It relies on the application to move on to the second IP address if the first one times out.
But I think the client DNS cache just stores 1 IP address, so you need a low TTL, because if one proxy goes down, clients that have that IP cached will lose access until they do a new DNS lookup. -
I've looked into it and the way I understand it is that it depends on your infrastructure.
If you have haproxy load balancers in active/passive failover and they are together in the same LAN, you can use a virtual (floating) ip address. Search for haproxy heartbeat.
-
Floating ip. But corosync is really an overkill. Vultr has a nice guide on floating ips.
-
Keepalived is most likely what you're looking for. You assign a VIP to your interface and it keeps a heartbeat between the systems. The VIP (floating IP) will move between systems if there is an issue.
Once that's set up, use Serf to update your HAProxy configs or Consul for automatic service discovery which HAProxy can read.
If you're using a cloud provider, I'd just use one of their provided load balancers.
-
@Emad-R said in how do you deal with SPOF with HAproxy:
Hi,
I wonder if this ever came across where you use HAproxy but wish to made it redundant and can sustain it going down. I know it rarely ever go down but what do you do if you want to make it redundant.
Use something like Azure Traffic Manager.
That is designed to do exactly what you seem to be looking for.
-
@stacksofplates said in how do you deal with SPOF with HAproxy:
Keepalived is most likely what you're looking for. You assign a VIP to your interface and it keeps a heartbeat between the systems. The VIP (floating IP) will move between systems if there is an issue.
Once that's set up, use Serf to update your HAProxy configs or Consul for automatic service discovery which HAProxy can read.
If you're using a cloud provider, I'd just use one of their provided load balancers.
+1 for serf but I can not find any doc about serf as a distributed config (as consul).
I know about serf as a discovery/alive tool only... -
@matteo-nunziati said in how do you deal with SPOF with HAproxy:
@stacksofplates said in how do you deal with SPOF with HAproxy:
Keepalived is most likely what you're looking for. You assign a VIP to your interface and it keeps a heartbeat between the systems. The VIP (floating IP) will move between systems if there is an issue.
Once that's set up, use Serf to update your HAProxy configs or Consul for automatic service discovery which HAProxy can read.
If you're using a cloud provider, I'd just use one of their provided load balancers.
+1 for serf but I can not find any doc about serf as a distributed config (as consul).
I know about serf as a discovery/alive tool only...When systems send their messages to the cluster they can run a script depending on the message. Their example repository has simple Bash scripts to update an HAProxy config.
https://github.com/hashicorp/serf/blob/master/demo/web-load-balancer/README.md
It's very very simple, but sometimes that's easier than setting up a full Consul cluster.
-
I know this is an old post but thought I'd reply anyway:
Yes for simple configs DNS/GSLB in front, or something like Amazon ELB with PROXY PROTOCOL support for IP transparency.But for anything complex like persistence with stick tables etc. then you need a pair of HAProxy nodes with a floating IP address:
You can use Keepalive which has already been mentioned (which lots of people like) and its very mature.
However we currently use Heartbeat from linux-ha.org - which is ancient and therefore pretty solid.But anoyingly it's designed more for two servers with one sharred storage and STONITH (Shoot The Other Node in the Head).
Rather than load balancers where going active/active is not a problem.AndyZaks PulseHA project (pulseha.com) is lightweight, fast and specifically design for load balancers.
He started the project because Heartbeat and Keepalived can be a right pain when they split brain (loose the heartbeat).We're currently helping him add the finishing touches to it for our next gen version of Loadbalancer.org appliance.
-
We use DNS health checks for this, Route53 and CLoudFlare have this but it comes at a n additional cost.
