How to do UDP load balancing in Kubernetes.

UDP load balancing in Kubernetes

Recently I to migrated a server application to Kubernetes whose interface was based on the Radius protocoll . The deployment looked quiet simple: A Pod with a service:

(all artifacts like Dockerfiles and deployments referenced in this article can be found here )

Odd results during testing

Testing this setup produced some odd results:

• Scaling server replicas down resulted in clients running into timeouts
• Scaling server replicas up resulted in uneven load distribution, load was not redistributed to new server replicas

After doing some research I discovered that UDP based load balancing of internal services does not work as I expected so I started looking for an alternative solution.

Nginx as an alternative

Nginx offers UDP load balancing support in its non-commercial edition which is easy to set up . There are two drawbacks though:

• First nginx needs to be compiled with the --with-stream option which the images on docker hub usually do not include.
• Second nginx needs to be configured to discover new replicas of the server.

Using a headless service for service discovery

Headless services offer exactly that:

You can use a headless Service to interface with other service discovery mechanisms, without being tied to Kubernetes’ implementation. You can create what are termed “headless” Services, by explicitly specifying “None” for the cluster IP (.spec.clusterIP).

The new topology puts the nginx and the headless service in between the clients and the server:

The deployment specification for the headless service in front of the server looks like this:

---
apiVersion: v1
kind: Service
metadata:
  name: udp-server-service-headless
spec:
  clusterIP: None
  ports:
    - name: udp
      port: 10002
      protocol: UDP
  selector:
    app: udp-server
  type: ClusterIP

The service makes sure that endpoints are created for each new server replica and dns resolvable:

$ kubectl get endpoints
NAME                          ENDPOINTS                             AGE
udp-server-service-headless   10.244.1.27:10002,10.244.1.61:10002   98m
$ kubectl scale deployment udp-server-deployment --replicas=3
deployment.apps/udp-server-deployment scaled
$ kubectl get endpoints
NAME                          ENDPOINTS                                               AGE
udp-server-service-headless   10.244.1.27:10002,10.244.1.61:10002,10.244.1.62:10002   100m
$
$ kubectl exec udp-nginx-deployment-86bb97f9d5-jtv27 -- nslookup udp-server-service-headless
Server:		10.96.0.10
Address:	10.96.0.10#53

Name:	udp-server-service-headless.default.svc.cluster.local
Address: 10.244.1.27
Name:	udp-server-service-headless.default.svc.cluster.local
Address: 10.244.1.61
Name:	udp-server-service-headless.default.svc.cluster.local
Address: 10.244.1.62

The commands above show that scaling up the server replicas by 1 creates a new endpoint that is resolvable inside the nginx container.

Resolve DNS names in nginx

The nginx configuration for udp load balancing is only a few lines long:

#user  nobody;
worker_processes  1;

events {
    worker_connections  1024;
}

stream {
    resolver dns-server ipv6=off valid=30s;
    
    upstream udp_servers {
        zone upstream_udp_servers_dynamic 64k;
        least_conn;
        server udp-server-service-headless:10002;
    }
    
    server {
        listen     5353 udp;
        proxy_pass udp_servers;
    }
}

the resolver directive tells nginx to resolve ip addresses for the servers from the dns server dns-server which points to the kubernetes internal dns server. The upstream directive has only one server entry that contains the dns name of the kubernetes service: udp-server-service-headless.
Using this configuration nginx will lookup all ip addresses for the fqdn udp-server-service-headless and load balance between them.

BUT the non-commercial version only resolves these names at startup. Whereas the commercial edition can be configured do this periodically . The server directive would just need the additional parameter resolve:

server udp-server-service-headless:10002 resolve;

So in order to get new replicas using the non-commercial edition I had to employ a little trick described below.

Config reload

When nginx is running it can be told to reload the configuration with the command /usr/local/sbin/nginx -s reload. So the solution in this case was to start a second process that would periodically executes that command.
So I created a simple script which does just that:

#!/bin/sh
while :
do
	sleep 30
	/usr/local/sbin/nginx -s reload
done

Using that script nginx reloads the configuration every 30 seconds and thus resolves any new server replicas.
To start a second process in the container I used supervisord and instead of starting nginx directly when the container starts, servisord is started.
The supervisord configuration looks like this:

[supervisord]
nodaemon=true

[program:reload-config]
command=/reload_config.sh
...

[program:nginx]
command=nginx -g 'daemon off;'
...

[eventlistener:nginx_exit]
command=/stop-supervisor.sh
events=PROCESS_STATE_STOPPED, PROCESS_STATE_EXITED, PROCESS_STATE_FATAL

I omitted some parts to make the configuration clearer. This configuration does the following:

• Start nginx and the script for reloading the configuration when the container starts
• Call the script stop-supervisor.sh when nginx stops to gracefully stop all processes

Load balancing that works

Using this nginx in a deployment with a single replica and a service in front does the load balancing correctly:

• No requests are being dropped when draining the server replicas
• No timeouts occur on the client when server replicas are scaled down
• Load is evenly distributed when server replicas are added

Conclusion

Kubernetes internal load balancing for UDP has some odd behavior. For reliable load balancing using an nginx with a headless service as the backend will provide consistent and reliable load balancing.
Discovering new server replicas requires additional effort when using the non-commercial version of nginx.

Back to Home