Transactional reconciles with flux

Multiple changes in a single reconcile

In one of the projects I’m working on, we have a fairly simple setup with fluxcd and helm . We only use two git repositroies: One for the Helm charts and one containing the HelmRelease and the GitRepository reference to the charts repository. The later git repository being synced by flux in the cluster. We chose this setup because we wanted to enable as many team members (with different technical backgrounds) of our DevOps team to contribute to our cloud deployment. Althoguh being simple we encountered an issue when changing the HelmRelease and the GitRepository in one commit resulting in two reconciles back to back.

The problem

Here is what the HelmRelease and GitRepository look like in the git repository that is synced by flux (containing only the relevant parts):

---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: test-app
spec:
  interval: 5m
  chart:
    spec:
      chart: test
      sourceRef:
        kind: GitRepository
        name: test-repo
      interval: 1m
  values:
    app-version: 1.0.1
---
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
  name: test-repo
spec:
  interval: 1m
  ref:
    tag: 1.2.2
  url: https://eyample.com/helm.git

When changing the app-version value of an image in the HelmRelease and the tag in the GitRepository in one commit, flux reconciles the HelmRelease first and then the GitRepository. This results in the HelmRelease being reconciled with the new app-version and the old tag from the GitRepository. That is because flux reconciles manfiests of a HelmRelase and GitRepository separately and immediately when they are changed. This can lead to failed reconciles if the new app-version of the image is not compatible with the old tag of the chart. In thses cases the first reconcile will fail and only the second reconcile will succeed. So how can we deal with situations where we want both changes being reconciled in one transaction / together ?

Soluton I: Putting HelmRelease and GitRepository in different git repositories

One solution is to put the HelmRelease and the GitRepository manifest in two different git repositories. That way updating the chart’s tag can be done first (which results in a reconcild of the GitRepository only) and then updating the app-version in the HelmRelease which will then use the new tag from the GitRepository, ensuring that both changes are reconciled together. This means adding a second git repository flux synces in the cluster. In addition the interval (the period the HelmRelease watches for changes) of the HelmRelease needs to be set to a high value in order to avoid the HelmRelease being reconciled in the time frame between the two commits. Since we did not want to add a second git repository to our setup, we searched for another solution.

Solution II: Suspending and resuming the HelmRelease

The solution we chose was to suspend the HelmRelease before doing both changes and resume its reconciliation after app-version and tag have been updated. This ensures that the HelmRelease will only be reconciled once with the new app-version and tag. In one transaction so to speak. There are two options to suspend and resume the HelmRelease:

1. Using the flux CLI:

flux suspend hr test-app

and

flux resume hr test-app

2. Using the HelmRelease manifest:

---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: test-app
spec:
  suspend: true # true to suspend, false to resume

Since the first approach requires the flux CLI and access to the cluster, it was unsuitable for us because most team members neither had experience with the CLI nor access to the cluster. So we decided to use the second approach which is declarative and can be done by anyone with access to the git repository (GitOps).

Conclusion

To ensure that changes to values (like app-version) in a HelmRelease are reconciled together with the referenced version of the helm chart, either two git repositories can be used or the HelmRelease can be suspended before the changes and then resumed. There is also a third solution and that is to keep the helm chart backwards compatible which might not be possible in all cases though.

Back to Home