阅读中文版本

Build a Rate Limiting Reverse Proxy in Node.js

A streaming reverse proxy that enforces token bucket and fixed window limits per API key before proxying, with standard 429 responses and Retry-After headers.

rate-limit-gateway/
├── src/
│   ├── server.js
│   ├── proxy.js
│   ├── limiters/
│   │   ├── tokenBucket.js
│   │   └── fixedWindow.js
│   └── config.js
├── test/
│   ├── limiters.test.js
│   └── proxy.test.js
├── config.json
└── package.json

class TokenBucket {
  constructor({ capacity, refillPerSec }) {
    this.capacity = capacity;
    this.rate = refillPerSec;
    this.tokens = new Map();
  }
  decide(key, cost = 1, now = Date.now()) {
    const st = this.tokens.get(key) || { tokens: this.capacity, last: now };
    st.tokens = Math.min(this.capacity, st.tokens + ((now - st.last) / 1000) * this.rate);
    st.last = now;
    if (st.tokens >= cost) {
      st.tokens -= cost;
      this.tokens.set(key, st);
      return { admit: true, remaining: Math.floor(st.tokens), retryAfterMs: 0 };
    }
    this.tokens.set(key, st);
    return { admit: false, remaining: 0, retryAfterMs: Math.ceil(((cost - st.tokens) / this.rate) * 1000) };
  }
}

const HOP_BY_HOP = new Set(['connection', 'keep-alive', 'transfer-encoding', 'upgrade', 'proxy-authenticate', 'proxy-authorization', 'te', 'trailers']);
function pickHeaders(headers) {
  const out = {};
  for (const [k, v] of Object.entries(headers)) {
    if (!HOP_BY_HOP.has(k.toLowerCase())) out[k] = Array.isArray(v) ? v.join(', ') : v;
  }
  return out;
}

const verdict = limiter.decide(key, route.cost);
res.setHeader('x-ratelimit-remaining', String(verdict.remaining));
if (!verdict.admit) {
  res.writeHead(429, { 'retry-after': String(Math.ceil(verdict.retryAfterMs / 1000)) });
  return res.end(JSON.stringify({ error: 'rate limited' }));
}
return proxy(req, res, route.upstream);

test('eleventh burst request gets 429 with retry-after', async () => {
  const gw = await startGateway({ limit: { capacity: 10, refillPerSec: 0 } });
  for (let i = 0; i < 10; i++) await get(gw, '/api?k=test');
  const blocked = await get(gw, '/api?k=test');
  expect(blocked.status).toBe(429);
  expect(Number(blocked.headers['retry-after'])).toBeGreaterThan(0);
});

Project overview

A rate limiting gateway sits between clients and an upstream service, turning an unbounded stream of requests into a policed one. The proxying itself is the easy half; the value lives in the policy half: which key the limit applies to, which algorithm matches the traffic shape, and what the client learns from each response.

This project builds the pipeline as three composable stages with a clean seam at the limiter interface — that seam is what later lets the same gateway run per-process limits in development and Redis-backed limits in production without touching the proxy code.

The finished service delivers five capabilities:

• Streaming proxying — request and response bodies pass through without buffering the full payload in memory.
• Token bucket and fixed window limiters, selectable per route through a JSON config.
• Limit headers on every proxied response, exposing remaining quota and reset time.
• 429 responses with Retry-After computed from the bucket refill state, with zero upstream traffic for rejected requests.
• A /stats endpoint reporting current utilization per key for dashboards.

The architecture is a pipeline: an HTTP server accepts the request, a limiter stage decides admit-or-reject from an in-memory store keyed by API key plus route, and a proxy stage pipes sockets to the upstream using Node streams. Limit state lives behind a swappable module interface so the in-memory version can be replaced with Redis without touching the pipeline.

Setup

Verify the prerequisites before starting: Node.js 20 LTS or later, comfort with Node streams and the http module basics, and an understanding of HTTP status codes and hop-by-hop headers.

The runtime footprint is deliberately the platform alone — no runtime dependencies, which is the point of building a gateway by hand:

Create the project and install:

mkdir rate-limit-gateway && cd rate-limit-gateway
npm init -y
npm install -D vitest@^2.1.0

Configure the environment in package.json — the server entry, the test script, and the gateway configured entirely by config.json in the working directory:

{
  "name": "rate-limit-gateway",
  "scripts": { "start": "node src/server.js", "test": "vitest run" }
}

The directory structure mirrors the pipeline: one module per limiter algorithm under limiters/, the proxy as its own module owning only socket plumbing, and config.js as the only module that reads the JSON.

Phase 1: Limiter modules

The limiter phase implements token bucket and fixed window behind one decide(key, cost) interface returning admit, remaining, and retryAfterMs.

// src/limiters/tokenBucket.js
export class TokenBucket {
  constructor({ capacity, refillPerSec }) {
    this.capacity = capacity;
    this.rate = refillPerSec;
    this.tokens = new Map();
  }

  decide(key, cost = 1, now = Date.now()) {
    const st = this.tokens.get(key) || { tokens: this.capacity, last: now };
    st.tokens = Math.min(this.capacity, st.tokens + ((now - st.last) / 1000) * this.rate);
    st.last = now;
    if (st.tokens >= cost) {
      st.tokens -= cost;
      this.tokens.set(key, st);
      return { admit: true, remaining: Math.floor(st.tokens), retryAfterMs: 0 };
    }
    this.tokens.set(key, st);
    return {
      admit: false, remaining: 0,
      retryAfterMs: Math.ceil(((cost - st.tokens) / this.rate) * 1000),
    };
  }
}

Why two algorithms behind one interface? Because the trade-off is real and worth demonstrating rather than asserting. Fixed windows are one counter and trivially explicable, but they admit a double burst at the boundary — a client can spend a full window’s quota at the end of one window and another at the start of the next. Token buckets match bursty API traffic better: a client saves up capacity, spends it, and refills continuously, and the refill math directly yields a Retry-After value — (cost - tokens) / rate is the seconds until admission becomes possible, which is why the rejection response can be precise instead of conservative.

The decide function returns admission plus remaining quota and retry delay in one call. Returning all three together is what keeps response headers and rejection responses consistent with each other, since they derive from the same verdict rather than two separate queries that could disagree between calls. And the now parameter is the testability seam: limiter tests never call Date.now directly, a five-minute refill gap is now + 300000, and deterministic clocks turn the hardest concurrency test into arithmetic.

Acceptance for this phase: bursts drain the bucket, refill recovers admission, and rejected calls receive a positive retryAfterMs that decreases over time.

Phase 2: Streaming proxy

The proxy pipes client requests to the upstream with the raw http.request API, forwarding method, headers, and body as streams while filtering hop-by-hop headers.

// src/proxy.js
import { request as httpRequest } from 'node:http';

const HOP_BY_HOP = new Set([
  'connection', 'keep-alive', 'transfer-encoding', 'upgrade',
  'proxy-authenticate', 'proxy-authorization', 'te', 'trailers',
]);

export function pickHeaders(headers) {
  const out = {};
  for (const [k, v] of Object.entries(headers)) {
    if (!HOP_BY_HOP.has(k.toLowerCase())) out[k] = Array.isArray(v) ? v.join(', ') : v;
  }
  return out;
}

export function proxy(req, res, upstream) {
  const upstreamReq = httpRequest(upstream, {
    method: req.method,
    headers: pickHeaders(req.headers),
    path: req.url,
  });
  req.pipe(upstreamReq);                       // client body -> upstream, streamed
  upstreamReq.on('response', (upstreamRes) => {
    res.writeHead(upstreamRes.statusCode, pickHeaders(upstreamRes.headers));
    upstreamRes.pipe(res);                     // upstream body -> client, streamed
  });
  upstreamReq.on('error', (err) => {
    res.writeHead(502).end(JSON.stringify({ error: 'upstream unreachable' }));
  });
}

Two properties are load-bearing. The proxy pipes, never buffers: the client socket pipes into the upstream request and the upstream response pipes back into the client socket, so a 100MB upload passes through with constant memory — and the test suite pins this with a heap-delta assertion, because the failure mode of accidentally accumulating a buffer and calling res.end(body) is silent until the first large file. Hop-by-hop headers are stripped in both directions: connection, transfer-encoding, and their siblings describe one TCP hop, not the request, and forwarding them corrupts the upstream connection — the filter is deliberately tiny and tested against the RFC list.

The 502 path is the honest error: an unreachable upstream answers with a clear status rather than hanging the client socket, and no retry happens here because the limiter verdict was already spent — retrying rejected or failed requests inside the proxy would silently double-count quota.

Acceptance for this phase: a proxied request arrives upstream with the original headers minus hop-by-hop ones, and response bodies of tens of megabytes stream through with flat memory.

Phase 3: Gateway pipeline

The server composes authentication, the route limiter, and the proxy: consult the limiter keyed by API key plus route, then proxy or answer 429 with Retry-After and quota headers.

// src/server.js
export function createHandler({ config, limiterFor, proxy }) {
  return (req, res) => {
    const key = keyFrom(req);                        // Authorization header or query
    const route = config.routeFor(req.url);
    const limiter = limiterFor(route);
    const verdict = limiter.decide(key + ':' + route.name, route.cost);

    res.setHeader('x-ratelimit-remaining', String(verdict.remaining));
    res.setHeader('x-ratelimit-limit', String(route.limit));

    if (!verdict.admit) {
      res.writeHead(429, { 'retry-after': String(Math.ceil(verdict.retryAfterMs / 1000)) });
      return res.end(JSON.stringify({ error: 'rate limited' }));
    }
    return proxy(req, res, route.upstream);
  };
}

The ordering inside the handler is the correctness contract: the limiter runs before proxy is invoked, so a rejected request generates zero upstream traffic — a gateway that proxies first and rejects after has already spent the upstream capacity it exists to protect. The quota headers are set on every response, admitted or rejected, so a well-behaved client can watch its remaining budget drain and back off before hitting the wall.

The limiter key is the API key plus route name, so one client’s budget on /api/search is independent of its budget on /api/reports when the config says so — and the config, not the code, decides which routes share state. Route config arrives as JSON: pattern, upstream, algorithm name, capacity, refill, and cost per request.

Acceptance for this phase: requests under the limit proxy normally; the request past the limit receives 429 with a positive Retry-After and no upstream traffic is generated.

Phase 4: End-to-end tests

The final phase runs the gateway against a stub upstream that echoes received headers and serves a configurable body size, asserting limit headers, 429 behavior, and streaming memory behavior.

// test/proxy.test.js
test('eleventh burst request gets 429 with retry-after', async () => {
  const gw = await startGateway({ limit: { capacity: 10, refillPerSec: 0 } });
  for (let i = 0; i < 10; i++) await get(gw, '/api?k=test');
  const blocked = await get(gw, '/api?k=test');
  expect(blocked.status).toBe(429);
  expect(Number(blocked.headers['retry-after'])).toBeGreaterThan(0);
});

test('32MB body streams through with bounded heap', async () => {
  const gw = await startGateway({ upstreamBodyBytes: 32 * 1024 * 1024 });
  const before = process.memoryUsage().heapUsed;
  const res = await get(gw, '/api?k=big');
  const received = await countBytes(res);
  expect(received).toBe(32 * 1024 * 1024);
  expect(process.memoryUsage().heapUsed - before).toBeLessThan(8 * 1024 * 1024);
});

The end-to-end tier boots the real gateway and a real stub upstream on ephemeral ports — no mocks at the HTTP layer, because the properties under test (header forwarding, socket streaming, status codes on the wire) only exist with real sockets. The stub echoes its received headers back in the response body, which turns header forwarding into a plain assertion instead of a debug session: parse the echo, compare against what the client sent minus the hop-by-hop set.

The memory test is the capstone: 32MB through the gateway with a heap delta under 8MB proves the pipe-based implementation survived contact with reality, and it fails with numbers rather than an OOM crash if anyone introduces a buffering shortcut.

Acceptance for this phase: the suite passes, including the large-body streaming test whose peak heap stays under a fixed budget, and the stats endpoint reflects the drained bucket.

Design trade-offs

In-memory state over Redis. In-memory state is faster, has zero operational dependencies, and is per-process — which means each replica enforces the full limit, the documented limitation. The decide interface exists precisely so the Redis swap is a module replacement rather than a rewrite, and per-replica capacity division is the documented interim answer.

Token bucket default over fixed window default. Fixed windows are simpler to explain and admit double bursts at boundaries; token buckets match API traffic and produce exact Retry-After values. Both ship because the trade-off is a demonstration, not a doctrine.

Piped sockets over fetch-based proxying. fetch is the modern client and buffers whole bodies in its default usage; raw http.request with pipe is the API that streams natively in both directions. The verbosity buys the memory property the tests assert.

Quota headers on every response over headers only on rejection. Clients that can watch their budget drain can avoid the 429 entirely, which is the outcome rate limiting actually wants — shaping traffic, not rejecting it.

Testing strategy

Run npx vitest. Limiter unit tests use the injected clock so refill math is deterministic to the millisecond — burst, drain, reject, advance the clock, admit — with the exact token counts asserted rather than approximate ranges. Proxy tests start a real gateway plus a stub upstream on ephemeral ports and cover header forwarding, hop-by-hop filtering, 429 semantics, Retry-After accuracy, and quota headers.

The streaming test proxies a 32MB body and asserts the process.memoryUsage heap delta stays bounded, proving the proxy streams instead of buffering — the one test that catches the class of bug invisible to every functional assertion. The stats endpoint gets its own assertion against the drained bucket from the burst test, which keeps the dashboard-facing contract honest without a second fixture.

Troubleshooting playbook

Upstream receives transfer-encoding or connection headers it rejects. The upstream 400s on headers the client never sent — or did send, about a different hop. The cause is hop-by-hop filtering missing from one direction: the filter must run on both the forwarded request and the returned response. The fix is extending the set in pickHeaders, and the echo-stub tests assert both directions against the RFC list.

Limiter admits slightly more than the configured burst. An eleventh request slips through a capacity of ten. The cause is clock mixing — one decide call reading Date.now while another reads a test-injected time, double-counting refill between them. The fix is passing a single monotonic clock instance to every call, and the deterministic limiter tests fail with an exact count the moment mixing reappears.

429 appears after the request already hit the upstream. Rejection and proxying both happen, upstream sees full traffic. The cause is ordering — proxy invoked before the verdict, or an error path inside the proxy retrying a rejected request. The fix is the handler’s sequence: limiter first, proxy only on admission, no retries inside the proxy, and the upstream-count assertion in the burst test fails with the excess request count.

Extending the project

The limiter interface is the growth surface: a Redis-backed decide over INCR with TTL is a new module in limiters/, selected by config, and the pipeline never learns the state moved. Sliding window log is a third algorithm with the same shape. Request cost weighting — charging more for expensive endpoints — is a cost field the config already carries. A circuit breaker is a proxy-stage concern, upstream-keyed rather than client-keyed, and sits cleanly in the same pipeline between limiter and proxy.

For deployment, run one or more replicas behind a TLS-terminating load balancer. Configure per-route limits in config.json mounted as a volume and reload with SIGHUP without dropping sockets. For multi-replica strict limits, implement the same decide interface over Redis INCR with TTL and swap the module in config — and scale per-replica capacity down by the replica count in the interim, which the known limitations document spells out. Choosing between the two shipped algorithms is a traffic-shape question with a concrete answer matrix. Token bucket fits traffic that legitimately bursts — interactive APIs where a user’s burst of activity should succeed — because saved capacity is a feature; the cost is that a client idle for an hour arrives with a full budget, so capacity is an upper bound on burst, not a steady-state rate. Fixed window fits traffic that should be flat — batch pollers, webhooks, cron-driven integrations — where the boundary double-burst is acceptable and the counter’s arithmetic is worth its simplicity. When neither shape fits because the traffic needs a strict rolling count, the sliding-window-log algorithm drops into the same interface: store the last N timestamps per key, prune on decide, and reject when the window is full — more memory per key, exact semantics, no boundary artifact. The config selecting per route which limiter runs is what turns this matrix into an operational decision rather than an architectural one.