Test probes using network policies - Mon, Mar 15, 2021
How to use network policies to test health probes
How to use network policies to test health probes
Recently I faced the problem how to test a health probe
of a container. To be more precise it was a liveness probe that would only succeed if the connection to its on premise database was working.
There are a lot of options on how to test this probe; Adding a temporary firewall rule, blocking traffic on the database, and so forth.
Since these options all require modifications to external systems I opted for a pure Kubernetes approach by putting a network policy
between the Pod and the database.
Setting up the network policy
Setting up the policy is straight forward. There some good examples for network policies in this repository
I used one that would generally block egress traffic from the Pod expect for DNS resolution requests:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: deny-external-egress
namespace: my-app-test
spec:
podSelector:
matchLabels:
app: my-app
policyTypes:
- Egress
egress:
- ports:
- port: 53
protocol: UDP
- port: 53
protocol: TCP
- to:
- namespaceSelector: {}
Since the postgres database listens on port 5432
that traffic would be blocked. The podSelector
makes sure that only the Pod my-app
is affected.
Test the test
Shortly after applying the network policy the Pods started to get unhealthy and Kubernetes tried to restart them. Corresponding events appeared as well:
$ kubectl get events -n my-app-test
my-app-test Warning Unhealthy pod/my-app-6c84dd4647-w5jjn Liveness probe failed: HTTP probe failed with statuscode: 503
my-app-test Normal Killing pod/my-app-6c84dd4647-w5jjn Container my-app failed liveness probe, will be restarted
my-app-test Warning BackOff pod/my-app-6c84dd4647-w5jjn Back-off restarting failed container
After removing the network policy, the Pods recovered and started to get healthy again.
Conclusion
Using network policies is an easy way to test liveness probes which evaluate connectivity to other components being it in cluster or on premise. The main advantage being that no external system needs to be altered to do the test.
Using the same approach resilience of an application to network outages can be tested.