Harden a Container with a Read-Only Root Filesystem

1. Prove the image starts with a read-only root and tmpfs mounts for its two writable paths before changing Kubernetes.

docker run --rm --read-only \
  --tmpfs /tmp:rw,noexec,nosuid,size=32m \
  --tmpfs /var/cache/nginx:rw,noexec,nosuid,size=64m \
  nginxinc/nginx-unprivileged:1.27-alpine nginx -t

Expected output: nginx reports that the configuration file /etc/nginx/nginx.conf syntax is ok and the test is successful.

2. Confirm that the image root is not writable while the tmpfs path still accepts writes.

docker run --rm --read-only \
  --tmpfs /tmp:rw,noexec,nosuid,size=32m \
  --tmpfs /var/cache/nginx:rw,noexec,nosuid,size=64m \
  nginxinc/nginx-unprivileged:1.27-alpine \
  sh -c 'touch /usr/share/nginx/html/probe 2>&1 || true; touch /tmp/ok && echo tmp-writable'

Expected output: touch reports Read-only file system for the image path and prints tmp-writable for /tmp.

3. Create the namespace and apply a Deployment with a read-only root, no capabilities, a non-root user, and emptyDir mounts for cache and temporary files.

kubectl create namespace hardening --dry-run=client -o yaml | kubectl apply -f -
kubectl apply -f k8s/nginx-readonly.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-readonly
  namespace: hardening
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx-readonly
  template:
    metadata:
      labels:
        app: nginx-readonly
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 101
        runAsGroup: 101
        fsGroup: 101
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: nginx
          image: nginxinc/nginx-unprivileged:1.27-alpine
          ports:
            - name: http
              containerPort: 8080
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
          readinessProbe:
            httpGet:
              path: /
              port: http
            initialDelaySeconds: 2
            periodSeconds: 5
          volumeMounts:
            - name: nginx-cache
              mountPath: /var/cache/nginx
            - name: nginx-tmp
              mountPath: /tmp
      volumes:
        - name: nginx-cache
          emptyDir:
            sizeLimit: 64Mi
        - name: nginx-tmp
          emptyDir:
            sizeLimit: 32Mi

Expected output: The namespace is configured and the Deployment is created; the manifest includes readOnlyRootFilesystem: true and capabilities.drop: [ALL].

4. Wait for the rollout and confirm both replicas are Running with zero restarts.

kubectl -n hardening rollout status deploy/nginx-readonly --timeout=90s
kubectl -n hardening get pods -l app=nginx-readonly

Expected output: The rollout completes and both pods show Running with 0 restarts after several minutes.

5. Inspect the effective container security context to prove the settings were not dropped by admission or the manifest.

kubectl -n hardening get pod -l app=nginx-readonly \
  -o jsonpath='{.items[0].spec.containers[0].securityContext}' | jq

Expected output: The JSON shows readOnlyRootFilesystem true, allowPrivilegeEscalation false, and capabilities.drop containing ALL.

6. Attempt a write to the image filesystem and a write to the mounted temporary directory from inside the running container.

kubectl -n hardening exec deploy/nginx-readonly -- \
  sh -c 'touch /usr/share/nginx/html/probe 2>&1 || true; touch /tmp/ok && echo tmp-writable'

Expected output: The image path reports Read-only file system and /tmp remains writable through the emptyDir mount.

7. Enforce the restricted Pod Security Standard on the namespace and restart the Deployment to prove new pods are admitted.

kubectl label namespace hardening pod-security.kubernetes.io/enforce=restricted --overwrite
kubectl -n hardening rollout restart deploy/nginx-readonly
kubectl -n hardening rollout status deploy/nginx-readonly --timeout=90s

Expected output: The namespace is labeled restricted and the replacement pods become Ready without an admission error.

Problem

The container crashes with mkdir() /var/cache/nginx/client_temp failed (30: Read-only file system)

Solution

nginx needs a writable cache directory. Mount an emptyDir at /var/cache/nginx, or configure all temp paths to a writable tmpfs.

Problem

The pod cannot write to an emptyDir despite the mount

Solution

Check that runAsUser, runAsGroup, and fsGroup allow the process to write. A mismatch between the image user and the volume group causes permission denied.

Problem

A custom image tries to write /var/run/nginx.pid or rewrite /etc/nginx

Solution

Use an unprivileged image that writes its pid to /tmp, or mount a small emptyDir for the specific runtime path instead of making the whole root writable.