How to use network policies to test health probes

How to use network policies to test health probes

Recently I faced the problem how to test a health probe of a container. To be more precise it was a liveness probe that would only succeed if the connection to its on premise database was working.
There are a lot of options on how to test this probe; Adding a temporary firewall rule, blocking traffic on the database, and so forth.
Since these options all require modifications to external systems I opted for a pure Kubernetes approach by putting a network policy between the Pod and the database.

Setting up the network policy

Setting up the policy is straight forward. There some good examples for network policies in this repository
I used one that would generally block egress traffic from the Pod expect for DNS resolution requests:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-external-egress
  namespace: my-app-test
spec:
  podSelector:
    matchLabels:
      app: my-app
  policyTypes:
  - Egress
  egress:
  - ports:
    - port: 53
      protocol: UDP
    - port: 53
      protocol: TCP
  - to:
    - namespaceSelector: {}

Since the postgres database listens on port 5432 that traffic would be blocked. The podSelector makes sure that only the Pod my-app is affected.

Test the test

Shortly after applying the network policy the Pods started to get unhealthy and Kubernetes tried to restart them. Corresponding events appeared as well:

$ kubectl get events -n my-app-test
my-app-test  Warning   Unhealthy         pod/my-app-6c84dd4647-w5jjn     Liveness probe failed: HTTP probe failed with statuscode: 503
my-app-test  Normal    Killing           pod/my-app-6c84dd4647-w5jjn  Container my-app failed liveness probe, will be restarted
my-app-test  Warning   BackOff           pod/my-app-6c84dd4647-w5jjn  Back-off restarting failed container

After removing the network policy, the Pods recovered and started to get healthy again.

Conclusion

Using network policies is an easy way to test liveness probes which evaluate connectivity to other components being it in cluster or on premise. The main advantage being that no external system needs to be altered to do the test.
Using the same approach resilience of an application to network outages can be tested.

Back to Home