Where to put CI/CD code - Sun, Jan 16, 2022

Where to put CI/CD code

How to deal with CI/CD code (Part 1)

Recently we discussed how to best deal with code used for CI/CD . Where ci code is the code needed for building and realizing the CI/CD pipelines like glue code or code for building the application.
Starting point for the discuss was that we all experienced that as systems grow, maintaining CI/CD code becomes more time consuming. Especially in a Microservice environment usually having a single git repository per service .
Problems that might be experienced may be:

So generally the quality of the code degraded. In this blog post I examine how putting CI/CD code into a dedicated repository can help maintain the quality of the code.
But first we take a look on how having the CI/CD code in the same repository looks like. Please note that this post is related to the usage of GitLab and it’s container registry

Keep CI code in the same repository

When you have a single project usually the CI/CD code is kept in the same repository. The usual place could be a CI or .ci folder. When using GitLab CI/CD pipelines , these files will be conveniently available in all the jobs of the pipeline. There are tow implications of doing so. Firstly code unrelated to the application itself resides in the same repository and secondly changes to the ci scripts might clutter up your git history.
An addition problem that you might face is that you need to find a suitable container for running your ci code. For example if you need additional tools like curl or more complex ones like the gcc compiler , you have to build your own ci images or install these tools in lengthy before-scripts lke the following:

before_script: |-
  yum install -y curl
  yum install -y devtoolset-8-binutils devtoolset-8-gcc-c++
  yum install -y epel-release
  yum install -y rpm-build pandoc
  ...

This approach might significantly increases ci job time. Also using dedicated ci images poses the question where to put and build these images. Especially if you only want to keep application images in the registry associated with your repository and keep the code for building ci images outside of the application’s ci.
Fortunately GitLab offers the possibility to separate different parts of the ci into different files through the usage of templates and Includes giving you the possibility to at separate ci code from application code in the same repository.
So having ci code in the same repository can make managing the project more complicated and may introduce additional code and images that are not really related to the application.

Putting CI code in a dedicated repository

This approach encompasses putting all ci related code and images into a different repository / image registry and using them during a CI/CD cycle by either including them as submodules , cloning them during a ci run or using the already mentioned GitLab Includes .
The main effect of this approach is that the ci code is separately managed and evolved from the application code, following the separation of concerns principle . This approach has the following benefits:

Especially the last two points can help ease the initially mentioned problem of degrading ci code quality. When similar ci code is used in multiple projects, this approach obviously scales best and yields the following additional benefits:

The main disadvantage is that an additional repository is needed. Although probably much smaller compared in size to the actual application repository.

Conclusion

Putting ci code in a dedicated repository has the benefits of separating the code and images by their concerns (app vs. ci), being able to do continuous integration on ci code as well (including quality checks) and having less ci code in application project. Note that application specific ci code (e.g. parts of test code) may still need to be put in the application repository.
The benefits of a dedicated repository are increased when having multiple projects using the same ci repository.

Back to Home