Docker multi stage builds - Tue, Jan 11, 2022
Using multi stage builds to reduce image size
Using multi state builds to reduce image size
Multi stage builds
are around since quiet a while in docker. In short they offer a way to reduce the size of a runtime image by allowing the usage of different images for building, then using only artifacts needed at runtime while discarding the rest (e.g. compiler).
An example would be building a binary using an sdk image, and after building copying only the binary to the runtime image.
This post shows an example how big these reductions can be.
Build without multi stages
In my recent cloud project, the k8s runtime uses Kyverno
to enforce some security related policies. For example “containers must be required to run as non-root users”
In order to test the conformance of the k8s resources as early as possible, I wanted to do these checks as part of a CI cycle . Thus gaining feedback much early as oppose to deployment to the cluster.
Therefore I needed the Kyvernso CLI in a docker image. The Dockerfile plain approach would look something like this:
FROM golang:alpine RUN apk add git make gcc libc-dev RUN git clone --depth 1 https://github.com/kyverno/kyverno RUN cd kyverno && make cli ENTRYPOINT ["/go/kyverno/cmd/cli/kubectl-kyverno/kyverno"] CMD ["--help"]
The base for the image is an alpine golang image augmented with the needed build tools, and the cli is build using
make. Building the image is straight forward and the resulting size is about
podman build --tag "kyverno-no-multi-stage-build" . ... podman images kyverno-no-multi-stage-build REPOSITORY TAG IMAGE ID CREATED SIZE localhost/kyverno-no-multi-stage-build latest 26c8c65a8400 7 minutes ago 2.91 GB
How come the size is that big ? First of all the base image alone is around
300 MB and adding the required build libs adds an additional
140 MB. In addition kyverno’s build does produce quiet a lot of artifacts having a size of
2.43 GB. The detailed layer size is shown below:
podman image tree kyverno-no-multi-stage-build Image ID: 26c8c65a8400 Tags: [localhost/kyverno-no-multi-stage-build:latest] Size: 2.909GB Image Layers ├── ID: 8d3ac3489996 Size: 5.866MB ├── ID: 0230d85b843e Size: 769kB ├── ID: 1afb07c068f3 Size: 2.56kB ├── ID: 337b50a59b37 Size: 318MB ├── ID: 369ac6dc732b Size: 3.072kB Top Layer of: [docker.io/library/golang:alpine] ├── ID: af50ae034e40 Size: 140.8MB ├── ID: 5854b398eff0 Size: 13.35MB └── ID: 3c07e807a13e Size: 2.431GB Top Layer of: [localhost/kyverno-no-multi-stage-build:latest]
While this image can be used perfectly, cleaning up build artifacts, only keeping those needed would of course reduce the size considerably. A cleaned up version would still be around
Build with multi stages
Creating an equivalent image using multiple stages is done by building the cli using the golang image and copying only the executable to the runtime image, a very small alpine:
FROM golang:alpine as builder RUN apk add git make gcc libc-dev RUN git clone --depth 1 https://github.com/kyverno/kyverno RUN cd kyverno && make cli FROM alpine WORKDIR /workdir COPY --from=builder /go/kyverno/cmd/cli/kubectl-kyverno/kyverno /usr/bin ENTRYPOINT ["kyverno"] CMD ["--help"]
The key points here are the two
FROM statements. The first one runs all following commands in an image containing all that is needed for building. After building however we switch to the alpine image using a second
FROM statement. And finally we only copy the binaries needed to the target image. We can see that the size of the image is much smaller:
podman build --tag "kyverno-multi-stage-build" . ... podman images kyverno-multi-stage-build REPOSITORY TAG IMAGE ID CREATED SIZE localhost/kyverno-multi-stage-build latest 86d0d669f0b1 19 seconds ago 128 MB
And the layer structure shows that the kyverno cli actually uses most of the size:
podman image tree kyverno-multi-stage-build Image ID: 86d0d669f0b1 Tags: [localhost/kyverno-multi-stage-build:latest] Size: 128.5MB Image Layers ├── ID: 8d3ac3489996 Size: 5.866MB ├── ID: 67a04a8023ad Size: 2.909MB ├── ID: a4fdf72de125 Size: 30.71MB ├── ID: 904d07a6808e Size: 4.608kB ├── ID: 3ec6f7d80887 Size: 8.869MB Top Layer of: [docker.io/library/python:alpine] └── ID: bca841cc4570 Size: 80.09MB Top Layer of: [localhost/kyverno-multi-stage-build:latest]
We have shown that image sizes can be reduced by using “Multi stage builds”. Size reduction and the effectiveness of this pattern mostly depend on the size of the build tools that need to be used for building.