Elasticsearch — high disk watermark blocks index allocation

Elasticsearch marks indices read-only because disk usage exceeds the high watermark, blocking writes to prevent corruption from a full disk.

intermediatedebuggingelasticsearchdisk

Symptoms

error: cluster_block_exception, index is read-only
Flood-stage disk watermark exceeded, indices marked as read-only
Writes fail with 403 Forbidden but reads succeed

Possible causes

Disk usage exceeds the flood-stage watermark (95% by default)
The cluster has insufficient disk capacity for the data volume
Old indices or large shards consume disk without cleanup

Diagnosis steps

Check disk usage per node

curl -s localhost:9200/_cat/allocation?v

Interpretation

Shows disk usage per node; identify nodes over the watermark.

Review watermark settings

curl -s localhost:9200/_cluster/settings?include_defaults

Interpretation

Shows the high and flood-stage watermark thresholds.

Root cause

Disk usage exceeds the flood-stage watermark, so Elasticsearch sets indices to read-only to prevent data corruption, blocking all write operations.

Fix

Free disk space by deleting old indices, reducing shard count, or increasing disk capacity. Then remove the read-only block with PUT _all/_settings {"index.blocks.read_only_allow_delete": null}.

Verify the fix

Write a test document and confirm it succeeds; verify disk usage is below the high watermark.

Symptom walkthrough

Elasticsearch marks indices read-only because disk usage exceeds the high watermark, blocking writes to prevent corruption from a full disk. The error appears in production but may not surface in local development because the conditions that trigger it depend on the deployment environment or runtime configuration. Understanding the full symptom set helps distinguish this issue from similar errors.

Diagnosis in practice

Follow the diagnosis steps in the frontmatter to isolate the cause. Each step includes a command and an interpretation of the output. The goal is to narrow down whether the issue is in the configuration, the runtime, or the application code. Reproduce the error in a minimal setup before applying a fix.

Root cause

Disk usage exceeds the flood-stage watermark, so Elasticsearch sets indices to read-only to prevent data corruption, blocking all write operations.

Fix

Free disk space by deleting old indices, reducing shard count, or increasing disk capacity. Then remove the read-only block with PUT _all/_settings {“index.blocks.read_only_allow_delete”: null}. After applying the fix, monitor the system to confirm the error does not recur under load or in different environments.

Verification

Write a test document and confirm it succeeds; verify disk usage is below the high watermark. If the verification passes, the fix is confirmed; if not, re-examine the diagnosis steps for missed contributing factors.

Diagnostic commands

Run these commands to reproduce and diagnose the issue:

curl -s localhost:9200/_cat/allocation?v
curl -s localhost:9200/_cluster/settings?include_defaults