Your Docker Images Are Leaking Secrets (And docker history Proves It)

A fintech startup pushed their production Docker image to a private ECR registry. Private. Locked down IAM policies. Everything by the book. Except one contractor had pull access, ran docker history --no-trunc, and found the Stripe production API key sitting right there in layer 4. The key had been "deleted" in layer 6 with RUN rm /app/.env.

That rm command did absolutely nothing useful from a security perspective.

Docker Layers Don't Forget

Every instruction in a Dockerfile creates a layer. Layers are additive. When you COPY .env /app/.env in one step and RUN rm /app/.env three steps later, the file is gone from the final filesystem view. But the layer where you copied it? Still there. Intact. Extractable.

Run this against any image:

docker history --no-trunc your-image:latest

You'll see every command that built it. Every COPY. Every ENV. Every ARG value. And if someone used docker save to export the image as a tar, they can literally unpack each layer and browse the filesystem at that point in time.

# extract all layers from an image
docker save your-image:latest -o image.tar
mkdir extracted && tar xf image.tar -C extracted
# now browse each layer's filesystem
# yeah, that "deleted" .env file is right there in the earlier layer

Teams discover this the hard way. Usually after a pentest report.

The ENV and ARG Trap

Copying files is the obvious mistake. The subtle one is ARG and ENV.

# people do this constantly
ARG DATABASE_URL
ARG API_SECRET
ENV DATABASE_URL=${DATABASE_URL}
ENV API_SECRET=${API_SECRET}

RUN npm run migrate

ARG values show up in docker history. Full plaintext. ENV values persist in the image metadata. Run docker inspect on the final image and scroll to the Env section. Everything is there.

Even if you unset the env var in a later layer, the layer that set it still contains it. And docker inspect on any intermediate image will show it. This is not a bug. Docker works exactly as designed. The design just happens to be terrible for secrets.

What About .dockerignore?

Good question. Wrong solution for this problem. A .dockerignore prevents files from entering the build context. If you have .env in your .dockerignore, Docker won't even send it to the daemon. But most teams need the secrets during build time (installing private npm packages, running migrations, pulling from private registries). So they add the file, use it, delete it. Which we just covered doesn't work.

Multi-Stage Builds: The Actual Fix

Multi-stage builds solve about 80% of this. The idea: use one stage to build (where secrets are needed), then copy only the build artifacts to a clean final stage.

# Stage 1: build (this image gets discarded)
FROM node:20-alpine AS builder
WORKDIR /app
COPY package*.json ./

# secret is only in this stage
ARG NPM_TOKEN
RUN echo "//registry.npmjs.org/:_authToken=${NPM_TOKEN}" > .npmrc
RUN npm ci
RUN rm .npmrc

COPY . .
RUN npm run build

# Stage 2: production (clean, no secrets)
FROM node:20-alpine
WORKDIR /app
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
COPY package*.json ./

EXPOSE 3000
CMD ["node", "dist/index.js"]

The builder stage has the NPM_TOKEN. The final image doesn't. When you push your-image:latest, only the final stage ships. Someone running docker history on it won't see the ARG instruction because it belonged to a different stage.

But.

If your CI caches intermediate stages (and many CI systems do), those builder images might still exist somewhere. GitHub Actions' docker/build-push-action with cache-to options, for example. Make sure you're not caching and pushing the builder stage to a registry anyone can access.

BuildKit Secrets: The Right Way Since 2019

Docker BuildKit (enabled by default since Docker 23.0) has a dedicated secrets mechanism that never writes secrets to any layer. Most teams still don't use it because the Dockerfile syntax looks slightly different and nobody reads changelogs.

# syntax=docker/dockerfile:1
FROM node:20-alpine
WORKDIR /app
COPY package*.json ./

# mount the secret at build time - it NEVER hits a layer
RUN --mount=type=secret,id=npm_token     NPM_TOKEN=$(cat /run/secrets/npm_token) &&     echo "//registry.npmjs.org/:_authToken=${NPM_TOKEN}" > .npmrc &&     npm ci &&     rm .npmrc

COPY . .
RUN npm run build

Build it like this:

DOCKER_BUILDKIT=1 docker build     --secret id=npm_token,src=./npm-token.txt     -t your-image:latest .

The secret is mounted as a tmpfs during that single RUN instruction. When the instruction finishes, it's gone. Not in a layer. Not in history. Not in metadata. Gone. Dive (the image inspection tool) won't find it. docker save won't contain it.

Why isn't everyone using this? Honestly, because half the Docker tutorials on the internet were written in 2017 and still show the ARG pattern. Stack Overflow answers from 2016 have thousands of upvotes recommending approaches that leak secrets. And teams copy what they find.

Runtime Secrets Are a Separate Problem

Build-time secrets and runtime secrets need different solutions. Everything above covers build time. For runtime:

Environment variables (via docker run -e or docker-compose) are visible to anyone who can run docker inspect on the container. If your orchestrator is Kubernetes, they show up in pod specs. Not great, but at least they're not baked into the image itself.

Docker Swarm secrets mount as files at /run/secrets/ inside the container. Encrypted at rest, only sent to nodes that need them. Decent if you're on Swarm.

Kubernetes secrets are base64 encoded by default. Not encrypted. echo "cGFzc3dvcmQ=" | base64 -d gives you "password." You need to enable encryption at rest in etcd or use something like Sealed Secrets, SOPS, or an external secrets operator pointing at Vault/AWS Secrets Manager.

The point: even if your image is clean, your deployment config might be leaking.

Scanning for This

Trivy, Grype, and Snyk container scanning focus primarily on CVEs in installed packages. They won't flag a hardcoded API key in layer 3. For that you need secret scanning tools that understand image layers.

Some options:

• Trufflehog can scan container images directly since v3.63 — it unpacks layers and runs entropy + regex detection
• ggshield (GitGuardian) has a ggshield secret scan docker command
• Dive won't find secrets automatically but lets you browse each layer's filesystem manually — useful for audits

The gap: most CI pipelines run vulnerability scanning on images but skip secret scanning entirely. Two different categories of tools solving two different problems. You need both.

And for the stuff these tools miss — hardcoded internal URLs, staging credentials disguised as config values, database connection strings in YAML files buried three directories deep — that requires someone (or something) actually reading the code. ScanMyCode.dev runs Docker security audits that check for secrets in layers, misconfigured base images, and privilege escalation paths. You get a report with exact locations, not just a list of CVE numbers.

Quick Checklist for Tomorrow Morning

Pull your production image. Run these commands. Fix whatever you find before lunch.

# check image metadata for env vars
docker inspect your-image:latest | jq '.[0].Config.Env'

# check build history for ARG values
docker history --no-trunc your-image:latest

# if you have dive installed
dive your-image:latest
# browse each layer, look for .env, credentials, keys, tokens

# scan for secrets across all layers
trufflehog docker --image your-image:latest

If any of those commands turn up something you don't want exposed: rebuild with multi-stage or BuildKit secrets. Rotate the leaked credentials. Today, not next sprint.

The image already in production with the baked-in secret? Every system that pulled it has a copy of that secret. Rotating the credential is the only fix. You can't un-push an image layer.

Stop Treating Images Like Throwaway Build Artifacts

Docker images are distribution packages. They get pulled, cached, copied, exported, and inspected by more people and systems than you think. Treat them like you'd treat a release binary that ships to customers — because functionally, that's what they are.

Build-time secrets belong in BuildKit mounts. Runtime secrets belong in your orchestrator's secret management. Nothing sensitive belongs in a layer, an ENV instruction, or an ARG default value. Ever.

Not sure what's hiding in your Docker images? Submit them for a Docker security audit and get a full layer-by-layer breakdown within 24 hours. Better to find it yourself than to explain it in an incident postmortem.