Elasticsearch yellow cluster — Unassigned Replica Shards Explained

Diagnose Elasticsearch yellow cluster health by inspecting unassigned replicas, allocation explain output, disk watermarks, and node roles before adding capacity.

intermediatedebuggingelasticsearchcluster-health

Symptoms

GET _cluster/health returns status yellow
number_of_pending_tasks and unassigned_shards stay above zero
Logs mention replica shards could not be allocated
Searches work but the cluster never turns green

Possible causes

A single-node cluster has replicas configured but no second node to host them
Disk watermarks or allocation filters exclude every other node
Replica allocation is disabled or a node lacks the data role
The index has more replicas than there are eligible data nodes

Diagnosis steps

Read cluster health and unassigned counts

curl -s 'localhost:9200/_cluster/health?pretty'

Interpretation

status yellow means all primary shards are active but at least one replica is unassigned. Red would mean a primary is missing, which is a different incident.

Ask why allocation failed

curl -s 'localhost:9200/_cluster/allocation/explain?pretty'

Interpretation

The allocate_explanation and deciders array name the exact blocker, such as disk watermark, node filters, or no eligible node. Read the first decider that returns a no decision.

List unassigned shards and their reasons

curl -s 'localhost:9200/_cat/shards?v&h=index,shard,prirep,state,unassigned.reason' | grep UNASSIGNED

Interpretation

reason REPLICA_ADDED means a replica was requested but cannot be placed; NODE_LEFT means the node holding it departed; ALLOCATION_FAILED means repeated attempts failed.

Check disk headroom and allocation settings

curl -s 'localhost:9200/_cat/allocation?v' && curl -s 'localhost:9200/_cluster/settings?include_defaults=true&flat_settings=true' | grep -E 'allocation|watermark'

Interpretation

Nodes above the low watermark refuse new shards. An exclude name or awareness attribute can also make every node ineligible even with free disk.

Root cause

Elasticsearch only turns green when every primary and replica shard is allocated. Yellow is the expected state when replicas exist but no eligible data node can host them: a single-node development cluster, a node that left without replacement, or allocation deciders that reject all candidates. Search and indexing continue because primaries are active, so the cluster looks healthy while resilience is reduced.

Fix

For a single-node cluster, set index.number_of_replicas to 0 and accept yellow as unavailable, or add a second data node. For multi-node clusters, free disk below the low watermark, remove stale allocation excludes, ensure candidate nodes have the data role, and retry failed allocations with POST _cluster/reroute?retry_failed=true. Align replica count with the number of eligible data nodes.

Risk note

Setting replicas to 0 removes all redundancy; a node loss then takes the index offline. Rerouting a primary manually can cause data loss if the original copy is still alive. Reducing watermark thresholds to force allocation risks disk exhaustion and shard failures.

Verify the fix

Re-run _cluster/health and confirm status is green, or document yellow as accepted for a deliberate single-node setup. Verify unassigned_shards is zero and _cat/shards shows STARTED for every primary and replica.

Yellow means replicas, not data loss

status: yellow means every primary shard is active and searchable, but at least one replica is unassigned. Data is available; redundancy is not. red is the serious state, because a primary is missing.

{
  "status": "yellow",
  "number_of_nodes": 1,
  "active_primary_shards": 12,
  "unassigned_shards": 12
}

A single-node cluster with one replica per index will always be yellow. That is expected, not a bug to chase.

Ask the allocator why

curl -s 'localhost:9200/_cluster/allocation/explain?pretty'
curl -s 'localhost:9200/_cat/shards?v&h=index,shard,prirep,state,unassigned.reason' | grep UNASSIGNED

allocation_explain lists each decider with a decision and a human-readable explanation. Common blockers are disk watermark exceeded, node does not match index allocation filtering, and no eligible node with the data role. _cat/allocation shows free disk per node, so you can confirm a watermark issue immediately.

Resolve the placement problem

For a deliberate single-node deployment:

curl -X PUT 'localhost:9200/*/_settings' -H 'Content-Type: application/json' \
  -d '{"index": {"number_of_replicas": 0}}'

For a real cluster, add a data node or free disk, clear excludes, and retry:

curl -X POST 'localhost:9200/_cluster/reroute?retry_failed=true'

Do not lower watermarks or manually move primaries just to make the status green.

Verify

Poll _cluster/health until status is green and unassigned_shards is 0, then confirm every shard is STARTED in _cat/shards. If you intentionally run one node, record the yellow status as accepted and alert on red or on unassigned primaries instead, so a real availability problem is not hidden by a familiar color.