Docker multi stage builds - Tue, Jan 11, 2022
Using multi stage builds to reduce image size
Using multi state builds to reduce image size
Multi stage builds
are around since quiet a while in docker. In short they offer a way to reduce the size of a runtime image by allowing the usage of different images for building, then using only artifacts needed at runtime while discarding the rest (e.g. compiler).
An example would be building a binary using an sdk image, and after building copying only the binary to the runtime image.
This post shows an example how big these reductions can be.
Build without multi stages
In my recent cloud project, the k8s runtime uses Kyverno
to enforce some security related policies. For example “containers must be required to run as non-root users”
.
In order to test the conformance of the k8s resources as early as possible, I wanted to do these checks as part of a CI cycle
. Thus gaining feedback much early as oppose to deployment to the cluster.
Therefore I needed the Kyvernso CLI
in a docker image. The Dockerfile plain approach would look something like this:
FROM golang:alpine
RUN apk add git make gcc libc-dev
RUN git clone --depth 1 https://github.com/kyverno/kyverno
RUN cd kyverno && make cli
ENTRYPOINT ["/go/kyverno/cmd/cli/kubectl-kyverno/kyverno"]
CMD ["--help"]
The base for the image is an alpine golang image augmented with the needed build tools, and the cli is build using make. Building the image is straight forward and the resulting size is about ~3GB:
podman build --tag "kyverno-no-multi-stage-build" .
...
podman images kyverno-no-multi-stage-build
REPOSITORY TAG IMAGE ID CREATED SIZE
localhost/kyverno-no-multi-stage-build latest 26c8c65a8400 7 minutes ago 2.91 GB
How come the size is that big ? First of all the base image alone is around 300 MB and adding the required build libs adds an additional 140 MB. In addition kyverno’s build does produce quiet a lot of artifacts having a size of 2.43 GB. The detailed layer size is shown below:
podman image tree kyverno-no-multi-stage-build
Image ID: 26c8c65a8400
Tags: [localhost/kyverno-no-multi-stage-build:latest]
Size: 2.909GB
Image Layers
├── ID: 8d3ac3489996 Size: 5.866MB
├── ID: 0230d85b843e Size: 769kB
├── ID: 1afb07c068f3 Size: 2.56kB
├── ID: 337b50a59b37 Size: 318MB
├── ID: 369ac6dc732b Size: 3.072kB Top Layer of: [docker.io/library/golang:alpine]
├── ID: af50ae034e40 Size: 140.8MB
├── ID: 5854b398eff0 Size: 13.35MB
└── ID: 3c07e807a13e Size: 2.431GB Top Layer of: [localhost/kyverno-no-multi-stage-build:latest]
While this image can be used perfectly, cleaning up build artifacts, only keeping those needed would of course reduce the size considerably. A cleaned up version would still be around 500 MB.
Build with multi stages
Creating an equivalent image using multiple stages is done by building the cli using the golang image and copying only the executable to the runtime image, a very small alpine:
FROM golang:alpine as builder
RUN apk add git make gcc libc-dev
RUN git clone --depth 1 https://github.com/kyverno/kyverno
RUN cd kyverno && make cli
FROM alpine
WORKDIR /workdir
COPY --from=builder /go/kyverno/cmd/cli/kubectl-kyverno/kyverno /usr/bin
ENTRYPOINT ["kyverno"]
CMD ["--help"]
The key points here are the two FROM statements. The first one runs all following commands in an image containing all that is needed for building. After building however we switch to the alpine image using a second FROM statement. And finally we only copy the binaries needed to the target image. We can see that the size of the image is much smaller:
podman build --tag "kyverno-multi-stage-build" .
...
podman images kyverno-multi-stage-build
REPOSITORY TAG IMAGE ID CREATED SIZE
localhost/kyverno-multi-stage-build latest 86d0d669f0b1 19 seconds ago 128 MB
And the layer structure shows that the kyverno cli actually uses most of the size:
podman image tree kyverno-multi-stage-build
Image ID: 86d0d669f0b1
Tags: [localhost/kyverno-multi-stage-build:latest]
Size: 128.5MB
Image Layers
├── ID: 8d3ac3489996 Size: 5.866MB
├── ID: 67a04a8023ad Size: 2.909MB
├── ID: a4fdf72de125 Size: 30.71MB
├── ID: 904d07a6808e Size: 4.608kB
├── ID: 3ec6f7d80887 Size: 8.869MB Top Layer of: [docker.io/library/python:alpine]
└── ID: bca841cc4570 Size: 80.09MB Top Layer of: [localhost/kyverno-multi-stage-build:latest]
Conclusion
We have shown that image sizes can be reduced by using “Multi stage builds”. Size reduction and the effectiveness of this pattern mostly depend on the size of the build tools that need to be used for building.