Docker Intermediate #2: Build Cache — BuildKit and Layer Ordering

If Basics #6 set up the “dependencies → code” ordering as the first cache button, this post stacks the BuildKit-era cache tools on top of that.

This post in the Docker Intermediate series:

• #1 Multi-stage builds and image slimming
• #2 Build cache — layer ordering ← this post
• #3 docker-compose basics — web + db
• #4 compose deep dive — depends_on, healthcheck, profiles
• #5 Environment variables and secrets
• #6 Logging and debugging

BuildKit is the default #

Docker has two generations of build engines: the legacy builder, and BuildKit. The difference is large.

On modern Docker (20.10+), BuildKit is the default, so the features above are available without extra setup. To force-enable it:

DOCKER_BUILDKIT=1 docker build -t myapp .

Or use buildx:

docker buildx build -t myapp .

buildx is the extended CLI on top of BuildKit. For a single build it behaves like docker build; for advanced features (multi-platform, cache export) buildx is more natural.

Every example in this post assumes BuildKit / buildx is on.

Layer cache — once more, deeper #

Basics introduced the rule of putting dependency copies above code copies. Let’s unpack it a step further:

this layer's cache key = previous layer digest
                      + the instruction itself (including variables)
                      + (for COPY/ADD) hash of the files being copied

If any one of those changes, the cache is invalidated. Keeping that in mind makes it much easier to track down why caches break.

Easy-to-miss cache invalidation cases #

FROM python:3.14-slim
ARG GIT_SHA            # An ARG here ends up in every later layer's cache key
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
ARG BUILD_TIME         # An ARG here only affects layers below
LABEL build-time=$BUILD_TIME

ARG enters the cache key of every layer after the line where it’s declared. Put a per-build value like BUILD_TIME near the top and every layer rebuilds every time. Place frequently-changing ARGs at the end.

ENV APP_VERSION=1.2.3   # Every layer below includes this in its cache key

Same for ENV. Don’t put rapidly-changing env vars high in the file.

--mount=type=cache — share caches across builds #

This is BuildKit’s biggest gift. Even when the layer cache breaks, the package manager’s download cache can survive.

npm #

# syntax=docker/dockerfile:1.7
FROM node:20-slim
WORKDIR /app
COPY package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm \
    npm ci
COPY . .

--mount=type=cache,target=/root/.npm means:

• For the duration of this RUN, mount a cache directory persisted across builds at /root/.npm
• The next build on the same host gets the same cache mounted again
• The cache itself is not baked into the image (no size impact)

If only package-lock.json changes, the RUN does re-execute, but the npm download cache (~/.npm/_cacache) is still alive — so it skips the network and finishes fast.

The first line # syntax=docker/dockerfile:1.7 tells BuildKit which Dockerfile syntax version to use. Idiomatic for safely using newer features (especially --mount).

pip #

# syntax=docker/dockerfile:1.7
FROM python:3.14-slim
WORKDIR /app
COPY requirements.txt .
RUN --mount=type=cache,target=/root/.cache/pip \
    pip install -r requirements.txt
COPY . .

Same idea. Note this is the opposite of the old idiom of --no-cache-dir — when using a mount cache, drop --no-cache-dir.

apt #

# syntax=docker/dockerfile:1.7
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
    --mount=type=cache,target=/var/lib/apt/lists,sharing=locked \
    apt-get update && \
    apt-get install -y --no-install-recommends curl

Mount apt’s two cache locations together. sharing=locked keeps another concurrently-built stage from touching the same cache — needed for tools like apt that aren’t safe under concurrent access.

Once apt caches are on a mount cache, the rm -rf /var/lib/apt/lists/* from Basics is no longer necessary — the cache lives outside the image.

Go module cache #

# syntax=docker/dockerfile:1.7
FROM golang:1.23 AS builder
WORKDIR /build
COPY go.mod go.sum ./
RUN --mount=type=cache,target=/go/pkg/mod \
    go mod download
COPY . .
RUN --mount=type=cache,target=/go/pkg/mod \
    --mount=type=cache,target=/root/.cache/go-build \
    CGO_ENABLED=0 go build -o myapp

Mounting both Go caches (/go/pkg/mod for modules, /root/.cache/go-build for compilation) makes rebuilds dramatically faster.

COPY --link — parallelizing the build #

The default COPY lays new files on top of the previous layer, so it has to wait until that previous layer is built. --link makes the COPY independent of the parent layer, giving BuildKit room to parallelize.

FROM node:20-slim AS deps
WORKDIR /app
COPY --link package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm npm ci

FROM node:20-slim
WORKDIR /app
COPY --link --from=deps /app/node_modules ./node_modules
COPY --link . .

The biggest perceptible wins are with multi-stage + large --from=other copies. There’s almost no reason not to use --link, so default to it in new Dockerfiles.

--mount=type=secret — keep secrets out of the image #

A way to use build-time secrets (e.g. private package registry tokens) without baking them into the image.

# syntax=docker/dockerfile:1.7
FROM python:3.14-slim
WORKDIR /app
RUN --mount=type=secret,id=pypi,target=/root/.pypirc \
    pip install --extra-index-url https://... my-private-pkg

Pass the secret on build:

docker build --secret id=pypi,src=$HOME/.pypirc -t myapp .

/root/.pypirc exists only during this RUN and doesn’t end up in any layer of the image. The old pattern of passing secrets via ARG or ENV leaks them via docker history — always use this mount instead.

RUN --mount=type=ssh — private Git repos #

For things like go get / pip install git+ssh://... that need SSH access during the build.

# syntax=docker/dockerfile:1.7
FROM python:3.14
RUN --mount=type=ssh \
    pip install git+ssh://git@github.com/myorg/private-repo.git

docker build --ssh default -t myapp .

No SSH key gets baked in; the host’s ssh-agent is borrowed only for the duration of the build.

External caches — --cache-from / --cache-to #

The caches we’ve seen so far only work on the same host doing the build. CI machines spin up fresh, runners are spread across hosts — that doesn’t help. External caches fill that gap.

Registry cache #

Treat the build cache itself as something you push to a registry, like an image.

docker buildx build \
  --cache-to=type=registry,ref=ghcr.io/curtis/myapp:cache,mode=max \
  --cache-from=type=registry,ref=ghcr.io/curtis/myapp:cache \
  -t ghcr.io/curtis/myapp:1.0 \
  --push .

• --cache-to — where to export this build’s cache
• --cache-from — where to import a cache before this build starts
• mode=max — cache all layers (default min keeps only result layers)

Even when CI starts on a fresh machine each time, the cache lives on the registry, and builds stay fast.

GitHub Actions cache #

Builds running on GHA can use the GHA cache backend directly:

- uses: docker/build-push-action@v5
  with:
    push: true
    tags: ghcr.io/curtis/myapp:${{ github.sha }}
    cache-from: type=gha
    cache-to: type=gha,mode=max

GHA cache is free (around 10GB per repo) and accessible quickly within the workflow. No external registry push needed — the easiest option.

inline cache #

The lightest version: embed cache metadata directly into the image.

docker buildx build \
  --cache-to=type=inline \
  --cache-from=ghcr.io/curtis/myapp:latest \
  -t ghcr.io/curtis/myapp:latest \
  --push .

No separate cache store needed. The image you --cache-from carries the cache info itself. Limitation: only mode=min is possible, so it can’t cache every intermediate stage in a multi-stage build. Best for simple single-stage / small projects.

Cache strategy at a glance #

--no-cache and forcing cache refresh #

When the cache is hanging on to stale results:

docker build --no-cache -t myapp .
docker build --no-cache-filter=builder -t myapp .   # only one stage

--no-cache-filter is a BuildKit feature that lets you pick which stage’s cache to ignore. Useful when you want dependencies cached but the build itself to start fresh.

Common reasons cache doesn’t work #

• No syntax= line — older syntax interpretation, mounts ignored
• ARG / ENV near the top — keys change every build
• COPY . . near the top — any code change invalidates everything
• CI machine starts fresh each run — without external cache (type=gha, type=registry) nothing survives
• BuildKit isn’t enabled — Docker Desktop has it on by default, but older CI environments need to be explicit
• --no-cache baked in somewhere — check Makefile / CI scripts

Wrap-up #

The picture from this post:

• BuildKit is the default builder. One line — # syntax=docker/dockerfile:1.7 — turns on new features.
• Layer cache key = previous layer + instruction + content of copied files. Push frequently-changing ARG/ENV/COPY toward the bottom.
• --mount=type=cache shares npm/pip/apt/Go caches across builds — package downloads stay fast even when layer cache breaks.
• COPY --link unlocks BuildKit’s parallelization — make it the default in new Dockerfiles.
• --mount=type=secret/ssh keeps secrets and SSH keys out of the image during builds.
• For distributed CI, pick an external cache — type=gha, type=registry, or type=inline based on environment.

In the next post (#3 docker-compose basics — web + db) we step from a single container into defining several containers in one file. Half of this series belongs to Compose.