Layer Environment Differences with Kustomize Overlays

Keep one shared base manifest and a thin overlay per environment, so staging and production differ only where they must.

~30 minkuberneteskustomizedeployment

Task

Remove copy-pasted per-environment manifests and make every environment-specific change explicit, reviewable, and reproducible.

When to use this

Three near-identical Deployments drift apart because staging was patched by hand; a resource limit added to production never reaches staging, so load tests run with the wrong memory profile.

Prerequisites

kubectl 1.27 or newer, which embeds kustomize
A base directory with the shared Deployment and Service
One namespace per environment

Steps

1. Declare the shared resources in the base kustomization.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml

Expected output: The base renders a complete workload with no environment-specific values.

2. Add a production overlay that sets the namespace, image digest, replica count, and a patch.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: production
resources:
  - ../../base
images:
  - name: ghcr.io/acme/payments
    newName: ghcr.io/acme/payments
    digest: sha256:6f1c2a9d4b7e8f0a1c3d5e7f9b2a4c6d8e0f1a3b5c7d9e1f3a5b7c9d1e3f5a7b
replicas:
  - name: payments
    count: 3
patches:
  - target:
      kind: Deployment
      name: payments
    patch: |-
      - op: add
        path: /spec/template/spec/containers/0/resources/limits/memory
        value: 512Mi

Expected output: The production overlay renders three replicas, the pinned digest, and the memory limit.

3. Render every overlay and diff staging against production to expose accidental differences.

for env in staging production; do kubectl kustomize overlays/$env > /tmp/$env.yaml; done; diff -u /tmp/staging.yaml /tmp/production.yaml

Expected output: The diff contains only intentional differences such as replicas, image, namespace, and resource requests.

4. Validate the overlay against the API server before applying it.

kubectl apply -k overlays/production --server-side --dry-run=server

Expected output: The server-side dry run succeeds and reports only the objects the overlay owns.

Troubleshooting

Problem

A strategic merge patch silently applies to nothing

Solution

The patch must match apiVersion, kind, and metadata.name after namePrefix is applied; verify the rendered name first.

Problem

Two overlays keep overwriting the same field

Solution

Move the shared value into the base and leave only the genuine per-environment difference in each overlay.

Limitations

Kustomize has no conditional logic; use Helm when a manifest must branch on computed values.
namePrefix changes resource names, so patches and external references must use the prefixed names.

One base, many thin overlays

Copy-pasted manifests fail quietly. A field added to production is never added to staging, so the difference surfaces only when a load test or an incident exercises the missing value. Kustomize makes the shared part singular and the differences explicit: a base holds the Deployment and Service, and each overlay declares only the namespace, image, replicas, and patches it needs. Reviewers then read a short overlay instead of diffing two long YAML files.

Overlays should stay readable

Keep patches small and targeted. A JSON patch that adds one resource limit is easy to review; a full copy of the Deployment inside an overlay is not. Prefer images with a digest over editing the container image in a patch, because the digest is what the registry and the admission policy will verify. When an overlay needs a field that the base already sets correctly, delete it from the overlay rather than repeating it.

images:
  - name: ghcr.io/acme/payments
    digest: sha256:6f1c2a9d4b7e8f0a1c3d5e7f9b2a4c6d8e0f1a3b5c7d9e1f3a5b7c9d1e3f5a7b
replicas:
  - name: payments
    count: 3

Verify the render, then apply

Render every overlay in CI and diff staging against production; the diff should contain only the differences you can explain. Apply with --server-side --dry-run=server first, because the dry run catches immutable-field changes and admission policy rejections that a local render cannot see. Archive the rendered YAML next to the pipeline run so an incident review can see exactly what was submitted.