Add Namespace Resource Quotas to Stop Noisy Neighbors

Bound CPU, memory, and object counts per namespace with ResourceQuota, require requests on every pod, and verify enforcement with a stress test.

~25 mindeploykubernetesoperations

Task

Cap namespace-level resource consumption so one team workload cannot starve the rest of the cluster.

When to use this

A batch job in the analytics namespace scheduled fifty unbounded pods and consumed the entire cluster memory. Every other team had services OOM-killed before anyone could react.

Prerequisites

A Kubernetes cluster with multiple team namespaces
Cluster authority to create ResourceQuota and LimitRange objects
A way to run a stress workload inside a test namespace

Steps

1. Set a hard cap on the namespace.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-quota
  namespace: analytics
spec:
  hard:
    requests.cpu: "8"
    requests.memory: 16Gi
    limits.memory: 24Gi
    pods: "40"

Expected output: The namespace can never request more than eight CPUs, sixteen GiB of memory, or run more than forty pods.

2. Require requests on every pod with a LimitRange.

apiVersion: v1
kind: LimitRange
metadata:
  name: defaults
  namespace: analytics
spec:
  limits:
    - type: Container
      defaultRequest: { cpu: 100m, memory: 128Mi }
      default: { cpu: "1", memory: 512Mi }

Expected output: Pods that omit requests get sane defaults instead of being scheduled as zero-cost squatters.

3. Verify the quota is visible and counting.

kubectl describe resourcequota team-quota -n analytics | head -12

Expected output: The describe output shows used versus hard for every resource, proving enforcement is live.

4. Stress the boundary with a deliberately oversized workload.

kubectl apply -f oversize-job.yaml -n analytics 2>&1 || kubectl get events -n analytics --field-selector reason=FailedCreate | tail -2

Expected output: The oversized job fails admission with a quota-exceeded message instead of consuming the cluster.

5. Confirm other namespaces are unaffected.

kubectl top pods -n payments --sort-by=memory | head -3

Expected output: Payments pods keep their memory and restart counts, demonstrating isolation in practice.

Troubleshooting

Problem

Deployments blocked by quota errors

Solution

The rolling update surge pushes usage over the hard cap; raise the quota by the surge headroom or lower the deployment maxSurge.

Problem

Pods scheduled but throttled to death

Solution

Limits are the throttle, requests are the schedule; if limits are far below needs, raise them per workload rather than loosening the namespace cap.

Problem

Team games the quota with tiny requests

Solution

Quotas count requests, not fairness; pair ResourceQuota with PriorityClass so burstable noise still yields to critical services under pressure.

Limitations

Quotas apply per namespace, not per user; a namespace shared by many services needs finer internal governance.
Object-count quotas can break operators that create many small resources; verify operator behavior before adding count limits.

Fences between neighbors

Without quotas, every pod competes for the whole cluster, and the loudest workload wins. A ResourceQuota turns each namespace into a bounded pasture: hard caps on requests, limits, and object counts that the API server enforces at admission time. The noisy analytics job now fails to schedule its fifty-first pod instead of OOM-killing the payments service.

spec:
  hard:
    requests.cpu: "8"
    requests.memory: 16Gi
    limits.memory: 24Gi
    pods: "40"

# And defaults so nothing schedules as zero-cost
spec:
  limits:
    - type: Container
      defaultRequest: { cpu: 100m, memory: 128Mi }

Quota at admission, not at runtime

The quota rejects work at the API server before it ever schedules, which is why it is reliable: there is no race window where usage exceeds the cap while enforcement catches up. The price is arithmetic: rolling updates need surge headroom inside the cap, or deploys fail on quota exactly when you least expect them.

Pair the fence with priority

Quotas bound what a namespace may take; PriorityClass decides who wins when the cluster itself is under pressure. Together they answer both questions: the analytics team cannot take more than their pasture, and if the cluster sweats, their batch yields to the payments service. Either alone leaves a failure mode open.