Inspect and Defragment etcd Safely

Check etcd database size and fragmentation, then defragment one member at a time with a backup first.

~40 minetcdkubernetesmaintenance

Task

Safely defragment etcd to reclaim space from a fragmented key-value store.

When to use this

The etcd database grows while object count stays stable, indicating fragmentation that risks slow operations.

Prerequisites

kubectl access to the control plane
etcdctl with the right endpoints
A backup strategy

Steps

1. Back up then check the endpoint status and size.

ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key endpoint status -w table

Expected output: The table shows database size and whether defragmentation is needed.

2. Defragment members one at a time.

ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key defrag

Expected output: The target member compacts its database and returns to normal, one member at a time.

Troubleshooting

Problem

Operations slow during defrag

Solution

Defragment one member at a time and never the leader first; watch etcd health between members.

Problem

Disk still full

Solution

After defrag, compact revisions too; defrag alone does not remove old revisions.

Limitations

Defrag blocks the member briefly.
Requires TLS credentials.
Does not reduce the number of revisions.

What this recipe does

Safely defragment etcd to reclaim space from a fragmented key-value store.

The etcd database grows while object count stays stable, indicating fragmentation that risks slow operations.

Prerequisites

kubectl access to the control plane
etcdctl with the right endpoints
A backup strategy

Procedure

Back up then check the endpoint status and size.

ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key endpoint status -w table

Expected result: The table shows database size and whether defragmentation is needed.

Defragment members one at a time.

ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key defrag

Expected result: The target member compacts its database and returns to normal, one member at a time.

How to confirm it worked

Run the commands above and confirm every expected result in order. If a step does not produce the stated output, use the troubleshooting notes before continuing. A passing run shows etcd database size shrinks after defrag and cluster health stays green throughout.

Troubleshooting

Operations slow during defrag — Defragment one member at a time and never the leader first; watch etcd health between members.

Disk still full — After defrag, compact revisions too; defrag alone does not remove old revisions.

Limitations

Defrag blocks the member briefly.
Requires TLS credentials.
Does not reduce the number of revisions.