阅读中文版本

Build a Gossip Cluster Discovery

A gossip protocol for cluster member discovery where each node periodically exchanges its member list with a random peer, converging to a consistent view without a central registry.

gossip-cluster-discovery/
├── src/gossip.ts
├── src/member.ts
├── src/failure.ts
├── tests/gossip.test.ts
└── package.json

members.get(self).heartbeat++; members.get(self).lastSeen = now;

const peer = randomPeer(); const remote = await exchange(peer, members); merge(remote);

if (now - m.lastSeen > TIMEOUT) m.state = "dead"; if (now - m.lastSeen > GRACE) members.delete(m.id);

const seedList = await contact(seed); merge(seedList); gossip();

Project overview

A cluster that relies on a central registry for membership has a single point of failure and a bootstrap problem: the registry must exist before the cluster, and the cluster cannot survive its loss. A gossip protocol distributes the membership so every node holds the list, and a new node learns the list by talking to any existing node. There is no registry, and the cluster survives the loss of any node.

The protocol is simple: each node periodically picks a random peer and exchanges its full member list. The merge takes the higher heartbeat per member, so newer information wins. After a few rounds, every node has the same list, because the random exchanges spread the newest entry like a rumor.

The heartbeat is the proof of life. Each node increments its own heartbeat on every tick and includes it in every exchange. A peer that receives a higher heartbeat updates its entry, and a peer that does not hear from a node for a timeout marks it dead. The heartbeat is monotonic, so a delayed message with an old heartbeat does not overwrite a newer one.

The failure detection is what keeps the list from growing with dead nodes. A node that stops gossiping stops advancing its heartbeat, and its peers mark it dead after a timeout and remove it after a grace period. The grace period is long enough for the death to propagate, so every node removes the dead member, not just the ones that noticed first.

The value of this project is the convergence, not the gossip. Exchanging lists is trivial; proving that random exchanges converge to a consistent view in a few rounds is the part worth learning, because that is what lets a cluster discover itself without a registry.

What you will build

The project delivers five capabilities, each driven by a test:

• Random peer exchange
• Convergence without a registry
• Failure detection via heartbeat
• Member list compaction
• Join and leave protocol

Each node holds a member list of (id, address, heartbeat) entries. A ticker picks a random peer and sends its full list; the peer merges by taking the higher heartbeat per id. A failure detector marks a member dead when its heartbeat stops advancing, and removes it after a timeout. A new node joins by contacting a seed and gossips its presence.

Architecture and data model

The member list is a map from node id to an entry with address, heartbeat, last-seen, and state. The local node increments its heartbeat on every gossip tick and updates its last-seen. The state is alive, dead, or removed, and the transitions are driven by the failure detector.

The random exchange picks a peer from the alive members and sends the full list over UDP. The peer merges the received list by taking the higher heartbeat per id, then sends its merged list back. The initiator merges the response the same way. This is a push-pull exchange, which converges faster than a push-only or pull-only.

The merge is the core of the convergence. For each id in the received list, if the received heartbeat is higher, the entry is updated. If the id is new, it is added. If the received heartbeat is lower, the local entry is kept. This is a last-writer-wins per id, where the writer is the heartbeat, and it converges because the heartbeat is monotonic.

The failure detector runs on every tick. For each member, if the last-seen is older than the timeout, the state is set to dead. If the last-seen is older than the grace period, the member is removed from the list. A dead member is kept in the list during the grace period so its death propagates, because a node that has not yet heard the death would re-add it from its own list.

The join via seed is the bootstrap. A new node that knows only the seed address contacts the seed, sends its own entry, and receives the seed member list. It merges the list and starts gossiping. Within a few rounds, every node knows the new member, because the seed spreads it.

The compaction is the list size management. The full list is sent on every exchange, which is fine for a small cluster but wasteful for a large one. A delta exchange that sends only the changed entries, or a bloom filter that summarizes the list, is a real improvement for large clusters but out of scope.

Prerequisites and setup

You need the following before you begin:

• Node.js 20 LTS or later
• A network transport (TCP or UDP)
• At least one seed node address

The dependencies cover the core needs:

• node:dgram (built-in) — UDP transport for gossip messages

pnpm init

pnpm add -D typescript tsx vitest

Keep secrets and configuration in the environment, and validate them at boot so a misconfigured service fails loudly instead of silently dropping work.

Directory structure

gossip-cluster-discovery/ is organized so each concern is testable in isolation:

gossip-cluster-discovery/
├── src/gossip.ts
├── src/member.ts
├── src/failure.ts
├── tests/gossip.test.ts
└── package.json

Phase 1: Member list

Hold a map of id to (address, heartbeat, last-seen) and increment the local heartbeat on each tick.

members.get(self).heartbeat++; members.get(self).lastSeen = now;

The heartbeat is the monotonic proof of life.

The phase is driven by a test that asserts the expected behavior, so the implementation is correct before the next phase builds on it.

Phase 2: Random exchange

Pick a random peer, send the full member list, and merge the response by higher heartbeat.

const peer = randomPeer(); const remote = await exchange(peer, members); merge(remote);

A merge takes the higher heartbeat per id, so newer info wins.

The phase is driven by a test that asserts the expected behavior, so the implementation is correct before the next phase builds on it.

Phase 3: Failure detection

Mark a member dead when its heartbeat has not advanced for a timeout, and remove it after a grace period.

if (now - m.lastSeen > TIMEOUT) m.state = "dead"; if (now - m.lastSeen > GRACE) members.delete(m.id);

A dead member is kept briefly so the death propagates, then removed.

The phase is driven by a test that asserts the expected behavior, so the implementation is correct before the next phase builds on it.

Phase 4: Join via seed

A new node contacts a seed, sends its entry, and receives the seed member list to bootstrap.

const seedList = await contact(seed); merge(seedList); gossip();

A seed is a known address; the new node does not need the full list to start.

The phase is driven by a test that asserts the expected behavior, so the implementation is correct before the next phase builds on it.

Testing

Start three nodes with one as seed, join the third via the seed, and assert all three converge to the same member list; stop a node and assert the others mark it dead then remove it; partition and heal, and assert convergence resumes.

The convergence test starts three nodes with one as seed, joins the third via the seed, and after a few gossip rounds asserts all three have the same member list. The join test starts a seed, then starts a new node with only the seed address, and asserts the new node learns the other members.

The failure test stops a node, waits past the timeout, and asserts the other nodes mark it dead, then past the grace period and assert they remove it. The heartbeat test increments the local heartbeat and asserts the exchange carries the new value, and a peer that receives it updates.

The merge test merges a list with a higher heartbeat for an existing id and asserts the entry updates, and a lower heartbeat and asserts the entry is kept. The partition test stops exchanges between two nodes, waits, and asserts they diverge, then resumes exchanges and asserts they converge.

The idempotence test merges the same list twice and asserts the member list is unchanged. The removal test removes a member and asserts a subsequent exchange from that member does not re-add it, if the grace period has passed.

The random-peer test asserts the peer selection is uniform over the alive members, so no member is consistently missed.

pnpm vitest run

Observability

Track the random peer exchange rate, the gossip resource usage, and the error count as the three primary signals for the gossip-cluster-discovery system. A sudden change in any of these is the signal that something has shifted, not necessarily broken but worth a look before it becomes an incident. Chart the convergence without a registry alongside the random peer exchange so a correlation is visible at a glance, because a rising convergence without a registry with a stable random peer exchange means the system is doing more work per unit, not that it is failing.

Log every member list transition with the context that triggered it, because an operator who sees the symptom but not the cause will guess wrong. The log entry should carry the gossip identifier, the input that caused the transition, and the outcome, so the operator can reconstruct what happened without re-running it. A log without context is a line of text; a log with context is a diagnosis.

Set alerts on the derived metrics, not the raw counters. A rate of change is more actionable than a total, because a total grows monotonically and never tells you when the growth accelerated. Alert on the random peer exchange error rate crossing a threshold, the gossip utilization exceeding a percentage, and the convergence without a registry latency exceeding the SLO. Each alert should have a runbook link, so the operator who receives it at three in the morning knows what to do, not just what is wrong.

The gossip-cluster-discovery system should expose its metrics in a format the existing dashboard can scrape, so the observability is additive, not a new tool. A metric that nobody reads is noise, so before adding one, confirm it has a consumer. The dashboard has three panels: throughput, error rate, and resource utilization, each with a threshold line that is the SLO, so a glance tells an operator whether the system is healthy without reading numbers.

Deployment and operations

Run one gossip instance per node, with at least one seed address configured. A new node that knows only the seed can bootstrap, because the seed gives it the full list. A cluster with no seed requires every node to know every other node at start, which defeats the purpose of discovery.

Choose the gossip interval from the convergence requirement. A one-second interval converges in a few seconds; a one-minute interval converges in a few minutes. The interval is a trade-off between convergence speed and network traffic, and the right value depends on the cluster size and the network.

Set the timeout and the grace period from the expected failure mode. A node that pauses for garbage collection should not be marked dead, so the timeout should be longer than the longest pause. The grace period should be long enough for the death to propagate, which is a few gossip rounds.

Monitor the member count and the dead count. A fluctuating member count means nodes are joining and leaving, which is normal; a growing dead count means the grace period is too long or nodes are failing faster than they are removed. Both are operator signals.

Do not use gossip for strong consistency. Gossip is eventually consistent, so a read of the member list may be stale. A system that needs a consistent membership at a point in time needs a consensus protocol, not gossip.

Security considerations

Treat every input to the gossip-cluster-discovery system as untrusted, because the random peer exchange path is the one place an external caller controls what the system processes. Validate the input shape before any gossip operation, and reject early with a clear error rather than letting a malformed input propagate into the member list phase where it causes a confusing failure that is hard to trace back to its source.

Keep secrets and configuration in the environment, never in the repository. The node:dgram credentials, the signing keys, and the service endpoints are all environment variables, read once at boot and validated, so a misconfigured gossip-cluster-discovery fails loudly at startup instead of silently misbehaving at runtime. A system that starts with a bad configuration and runs anyway is a system that will produce wrong results for hours before anyone notices.

Rate-limit the random peer exchange endpoint per caller, because an unbounded gossip operation is a resource leak as much as a security problem. A caller that can trigger the random exchange path without limit can exhaust the system, whether intentionally or not. The rate limit is a per-caller ceiling that costs legitimate callers nothing and stops the abuse before it reaches the member list logic.

Audit-log every state-changing operation with the caller identity, the input, and the timestamp. The audit log is not for debugging, it is for accountability: if something went wrong, the log says who did what and when. The gossip-cluster-discovery system writes the audit log to an append-only store that the system itself cannot modify, so a compromised system cannot rewrite its history.

Do not expose internal errors to the caller. A stack trace is a map of the system internals, and an attacker who collects them learns the gossip structure. Return a generic error with a correlation id, and log the full error internally with the same id, so an operator can trace it without leaking it.

Limitations and next steps

• Gossip is eventually consistent; a join takes a few rounds to propagate.
• No security; a rogue node can join and poison the list.
• The member list is full-exchange; for large clusters, use a delta or a bloom filter.

Gossip is eventually consistent. A join takes a few rounds to propagate, so a node that joins and immediately asks for the member list may not see itself. This is fine for discovery, where the list is a hint, not a guarantee, but it is wrong for a system that needs a consistent view.

There is no security. A rogue node that knows the seed address can join and poison the member list with fake entries. A secure cluster needs authentication and authorization, which are out of scope, because gossip is a discovery protocol, not a security protocol.

The member list is full-exchange. Every gossip round sends the full list, which is O(n) per round per node, and O(n^2) per round for the cluster. For a large cluster, a delta exchange or a bloom filter reduces the traffic, but that is a different protocol.