HAProxy 503 no server available — Fix Failed Backend Health Checks

Diagnose HAProxy 503 errors by reading the runtime stats socket, identifying DOWN servers and check failures, then correcting health check probes and timeouts.

intermediatedebugginghaproxyload-balancing

Symptoms

HAProxy returns 503 Service Unavailable for every request to one backend
Logs show 'no server is available to handle this request'
The application responds on its port when tested directly from the HAProxy host
All servers in a backend flip DOWN together after a deploy

Possible causes

The health check path or port does not match what the application serves
The application requires TLS or a Host header that the check does not send
check inter, rise, and fall values are too aggressive for the startup time
Backend servers are at maxconn or removed by a configuration reload

Diagnosis steps

Query the runtime stats for server state

echo "show stat" | socat stdio /var/run/haproxy/admin.sock | cut -d, -f1,2,18,37,38 | column -s, -t | grep -E 'pxname|BACKEND|DOWN|UP'

Interpretation

A server with status DOWN and check_status L7STS or L4CON shows why HAProxy removed it. All servers DOWN in one backend explains the 503 immediately.

Show the detailed server state

echo "show servers state" | socat stdio /var/run/haproxy/admin.sock

Interpretation

The output includes the admin state, operational state, and check result per server. Compare it with the configuration to see whether a server was put into maintenance.

Reproduce the health check from the HAProxy host

curl -v -H 'Host: example.com' --max-time 3 http://10.0.0.20:8080/healthz

Interpretation

If this fails, HAProxy is reporting the truth. If it succeeds, the check is using a different port, path, method, or header than you tested.

Validate the configuration before reload

haproxy -c -f /etc/haproxy/haproxy.cfg && echo "show errors" | socat stdio /var/run/haproxy/admin.sock | tail -30

Interpretation

A config error prevents a reload from taking effect, leaving old servers in the backend. Runtime errors show recent connection and check failures.

Root cause

HAProxy only routes to servers it considers UP. A server becomes DOWN when its health check fails for fall consecutive attempts, so a 503 often means the probe, not the application, is wrong. Mismatched ports, paths, expected status codes, or TLS requirements mark healthy backends as unavailable, and an empty or maintenance-mode backend produces the same response.

Fix

Align option httpchk with a real endpoint, set the check port explicitly, and relax inter, rise, and fall for startup and GC pauses. Use http-check expect status 200 so redirects and auth pages are not accepted as healthy. Remove servers from maintenance and confirm maxconn is not exhausted. Validate the config and reload gracefully with -sf so existing connections drain.

Risk note

Loosening checks too far keeps genuinely broken servers in rotation. Backend servers removed by a config reload are unavailable until the new process takes over, so test with a canary first. Never point checks at an endpoint that depends on the database being writable unless that dependency is intentional.

Verify the fix

Run show stat and confirm every intended server is UP with check_status L7OK or L4OK. Send repeated requests through the frontend and confirm 200 responses with no 503, then take one server down and verify HAProxy drains it without failing requests.

503 usually means “no UP server”

HAProxy answers 503 from its own error page when a frontend accepts a request but the selected backend has no available server. The application may be perfectly healthy; HAProxy simply removed it from rotation.

<NOSRV> backend 'api' has no server available!

Read the runtime state

echo "show stat" | socat stdio /var/run/haproxy/admin.sock \
  | cut -d, -f1,2,18,37,38 | column -s, -t | head -20

Columns are proxy, server, status, check status, and check code. DOWN with L7STS means the probe got an HTTP status outside the accepted range. L4CON means the TCP connect failed. MAINT means an operator or automation disabled the server.

Compare the probe to reality

curl -v -H 'Host: example.com' --max-time 3 http://10.0.0.20:8080/healthz

If this returns 200 while HAProxy marks the server DOWN, the check differs: wrong port, wrong path, wrong Host header, or a TLS requirement. Fix the definition rather than disabling checks.

backend api
    option httpchk GET /healthz HTTP/1.1\r\nHost:\ example.com
    http-check expect status 200
    default-server inter 3s fall 3 rise 2
    server api-1 10.0.0.20:8080 check
    server api-2 10.0.0.21:8080 check

Reload and verify

Validate with haproxy -c -f /etc/haproxy/haproxy.cfg, then reload with -sf $(pidof haproxy) so old workers drain. Watch show stat until every server is UP, send a burst of requests through the frontend, and confirm no 503. Finally, stop one backend deliberately: HAProxy should mark it DOWN after fall checks and continue serving from the other without user-visible errors.