secure k8 s

Pinning images by digest: what I learned the multi-arch way

18. Mai 2026 · supply-chain · docker · image-digest · multi-arch

Standard advice for production container images is to pin them by digest, not tag. nginx:1.27-alpine is a moving target; nginx:1.27-alpine@sha256:65e3e85d... is the bytes you tested. I knew this. I had digest-pinned the base images in the FROM lines of the Dockerfile for this site. And I still shipped an image my cluster couldn’t pull, on the very first deploy, because there’s a quieter sharp edge a layer deeper than the one the advice covers.

This is the post-mortem on a thirty-minute distraction that pointed at a real problem with how digests, tags, and multi-arch interact.

The failure

I built the image locally on a Mac (arm64), pushed it to Docker Hub as docker.io/nyrvex/securek8s:bd21295, and ran helm install. The Pods went into ErrImagePull and then ImagePullBackOff:

$ kubectl -n securek8s describe pod -l app.kubernetes.io/instance=web | grep -A1 Failed
  Warning  Failed  Failed to pull image "docker.io/nyrvex/securek8s:bd21295":
                  rpc error: code = NotFound desc = failed to pull and unpack image
                  "docker.io/nyrvex/securek8s:bd21295":
                  no match for platform in manifest: not found

“No match for platform” is the giveaway. The image existed. The registry accepted the pull. The Hetzner workers, which are amd64, just couldn’t find an amd64 variant inside the manifest. I had pushed arm64-only and the cluster wasn’t arm64.

docker build on a Mac defaults to the host platform. There is no warning when you push such an image to a registry. The error only surfaces when something that isn’t arm64 tries to pull it.

The fix is one flag and a different builder:

docker buildx create --use --name multiarch
docker buildx build \
  --platform linux/amd64,linux/arm64 \
  -t docker.io/nyrvex/securek8s:bd21295 \
  -t docker.io/nyrvex/securek8s:latest \
  --push .

This produces a manifest list (a single tag that points at one manifest per architecture), not a single-arch image. After re-pushing, the cluster pulled it on the next reconcile and the deployment rolled out.

But the same incident raised a question worth a longer answer.

Tags are mutable. Digests are not. Manifest lists are both.

The classical case for digest pinning is straightforward:

  • nginx:1.27 is mutable. Today it points at sha256:abc...; tomorrow it points at sha256:def.... Your Dockerfile’s FROM nginx:1.27 will silently build against a different base.
  • nginx:1.27@sha256:abc... is immutable. The digest is a content hash; the bytes cannot change without the digest changing.

So far so good. The textbook fix is to pin every FROM line by digest, and I do.

The wrinkle is that for a multi-arch image, the digest in the manifest list is the digest of the manifest list, not of any particular architecture’s manifest. The manifest list looks something like:

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "manifests": [
    {"digest": "sha256:111... amd64...", "platform": {"architecture": "amd64", "os": "linux"}},
    {"digest": "sha256:222... arm64...", "platform": {"architecture": "arm64", "os": "linux"}}
  ]
}

Pull by the manifest-list digest, and the runtime picks the entry that matches its platform. Pull by an architecture-specific digest, and you get exactly those bytes — which is what you want for reproducibility, but means you have to know which arch you’re running.

The practical consequence is in FROM lines and in Kubernetes manifests:

# Pinned to the manifest list digest. The runtime picks amd64 or arm64.
FROM nginxinc/nginx-unprivileged:1.27-alpine@sha256:65e3e85d...
# Same idea: this is the manifest-list digest. K8s pulls the right arch per node.
image: docker.io/nyrvex/securek8s@sha256:c3f3dd26...

That works for a multi-arch cluster. It would not work if I’d pinned an arch-specific digest by mistake, which is a thing tools occasionally do when they read a manifest list and surface “the digest” of the inner manifest. The test is to inspect explicitly:

docker buildx imagetools inspect docker.io/nyrvex/securek8s:latest

If the output starts with MediaType: application/vnd.oci.image.index.v1+json, it’s a manifest list. If it starts with application/vnd.oci.image.manifest.v1+json, it’s a single-arch image masquerading as a tag.

The cost of pinning

Pinning is not free. Two costs are worth being explicit about.

1. You opt out of upstream security updates. If nginx-unprivileged:1.27-alpine ships a CVE fix tomorrow, your image will not absorb it on the next build until you lift the digest. That’s the point — it’s a feature, not a bug — but it means digest pinning forces base-image updates to become a deliberate process step. If you don’t have a process, you have a worse posture than the floating- tag people: you’re pinned to the version with the unpatched CVE.

2. Multi-arch builds are slow and need a builder. A buildx multi-arch build emulates the non-native architecture under QEMU (unless you have native builders of each arch). Build times triple. CI runners cost more. There are projects (Depot, BuildKit cluster) that fix this with native builders per arch, but for a one-person homelab, “buildx build with QEMU” is the realistic answer and the cost is felt.

What I should have done from day one

In hindsight, the bug was preventable. The first push was:

docker build -t docker.io/nyrvex/securek8s:bd21295 .  # arm64 only on a Mac
docker push  docker.io/nyrvex/securek8s:bd21295

A more cautious script would have failed loud the moment a single-arch image was pushed to a tag that’s expected to be multi-arch. Something like:

TAG=$(git rev-parse --short HEAD)
docker buildx build \
  --platform linux/amd64,linux/arm64 \
  -t docker.io/nyrvex/securek8s:$TAG \
  --push .

# Sanity check: the tag must be a manifest list, not a single-arch image.
docker buildx imagetools inspect docker.io/nyrvex/securek8s:$TAG \
  | grep -q 'image.index' \
  || { echo "ERROR: $TAG is not a manifest list" >&2; exit 1; }

That imagetools inspect line is the one I’d carry forward. It costs nothing and would have caught the exact bug the first time.

What I’d remember

  • Tags are mutable, single-arch digests are immutable, manifest-list digests are immutable but architecture-agnostic. The last is the one you usually want in Kubernetes.
  • “Build on Mac, deploy to amd64” is a daily failure mode if you don’t use --platform explicitly. The error message is helpful only if you’ve seen it before.
  • Pinning shifts when CVEs reach you. It doesn’t make them go away. If you pin without a process, you’ve made things worse.
  • docker buildx imagetools inspect is the underused tool for verifying what you actually pushed.