Tune Ingress Timeouts for Slow Endpoints

Raise proxy read and send timeouts only for routes that need them, and keep the default short everywhere else.

~25 minkubernetesingresstimeouts

Task

Stop 504 responses on legitimate long-running endpoints without letting every request hold an ingress connection for minutes.

When to use this

A report export endpoint takes 90 seconds to generate a PDF. The ingress default read timeout is 60 seconds, so every export fails with 504 even though the backend completes the work and writes the file.

Prerequisites

ingress-nginx installed in the cluster
An Ingress resource for the affected service
kubectl access to inspect the generated NGINX configuration

Steps

1. Annotate only the slow route with longer read and send timeouts, and keep connect timeouts short.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: reports-api
  annotations:
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "5"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "120"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "120"
    nginx.ingress.kubernetes.io/proxy-next-upstream: "error timeout http_502 http_503"
spec:
  ingressClassName: nginx
  rules:
    - host: api.example.com
      http:
        paths:
          - path: /reports/export
            pathType: Prefix
            backend:
              service:
                name: reports-api
                port:
                  number: 80

Expected output: Only /reports/export gets a 120 second read timeout; all other routes keep the controller default.

2. Confirm the generated configuration contains the override for the route.

kubectl exec deploy/ingress-nginx-controller -n ingress-nginx -- grep -A 6 'location /reports/export' /etc/nginx/nginx.conf

Expected output: The location block contains proxy_read_timeout 120s and proxy_send_timeout 120s.

3. Load test the slow endpoint and confirm it returns 200 rather than 504.

hey -n 5 -c 1 -t 180 https://api.example.com/reports/export

Expected output: All five requests return 200 with response times around 90 seconds.

4. Check the upstream service and load balancer timeouts so the ingress is not the longest link.

kubectl get svc reports-api -o jsonpath='{.metadata.annotations}'

Expected output: The service annotation or cloud load balancer idle timeout is at least as long as the ingress read timeout.

Troubleshooting

Problem

The annotation has no effect

Solution

Confirm the Ingress uses the nginx class and the controller version supports the annotation; a default backend or another controller may handle the route.

Problem

504 persists at exactly 60 seconds

Solution

Another hop, such as a cloud load balancer or service mesh, is timing out first; align timeouts across every hop in the path.

Limitations

Long timeouts consume ingress worker connections and reduce capacity for other requests.
Timeouts above a few minutes should usually be replaced by an asynchronous job with polling.

Timeouts are per hop

A 504 means one proxy stopped waiting before the backend answered. In a typical path the client, cloud load balancer, ingress controller, service mesh, and application each have their own timeout. Raising only the ingress moves the failure to the next shortest hop. Map every timeout in the path before changing anything, and make the outermost hop the largest.

Scope the override

Do not raise the default for the whole ingress class. A long default lets slow or malicious clients hold worker connections, reducing capacity for everyone. Apply annotations to the specific Ingress that needs them, keep proxy-connect-timeout short, and use proxy-next-upstream to retry safe failures against another replica.

nginx.ingress.kubernetes.io/proxy-read-timeout: "120"
nginx.ingress.kubernetes.io/proxy-send-timeout: "120"

Verify both directions

Read timeout covers the time between successive reads from the upstream; send timeout covers writes. A streaming response can trip the read timeout if no bytes flow for too long, even if the total duration is acceptable. Load test with the real payload and measure at the client. Also confirm the ingress controller reloaded; a bad annotation is ignored and the default silently applies.

Prefer async for long work

Beyond a minute or two, a synchronous request is fragile: clients disconnect, deploys restart pods, and connections consume capacity. Move the work to a job queue, return 202 with a job id, and let the client poll or receive a webhook. Keep the synchronous route only for work that reliably finishes inside the budget.