Sharding
Sharding partitions data across independent databases by a shard key, trading cross-shard simplicity for horizontal write capacity and operational complexity.
Definition
Sharding horizontally partitions a dataset across multiple databases or nodes. Each shard holds a subset of rows determined by a shard key, such as a tenant id or user id. Queries that include the shard key route to one shard; queries that span shards require scatter-gather or a separate analytical store.
Context and scope
Sharding is used when a single primary cannot handle write volume, storage, or isolation requirements. It is a late-stage scaling technique: indexing, caching, read replicas, and archiving usually solve capacity problems with far less complexity.
Common misconceptions
- Sharding automatically improves every query — cross-shard queries become slower and more complex
- Choosing a shard key is easy to change later — resharding is a major migration
- More shards always mean more availability — each shard is another failure domain and operational surface
Anti-patterns
- Sharding before measuring a single-node bottleneck
- Choosing a low-cardinality or hot shard key that concentrates traffic
- Using a random key that makes common queries scatter across every shard
Related concepts
Related examples
Authoritative source
Sharding is a data-model decision
A shard key determines which queries are cheap. If the key is tenant_id, every tenant-scoped query routes to one shard, but a global report must query every shard. If the key is user_id, user-scoped queries are local, but a tenant dashboard may scatter. Choose the key from the dominant access pattern and accept that some queries need a different store.
shard = hash(tenant_id) % shard_count
-- or a directory service that maps tenant_id -> shardHash-based routing spreads load but makes range queries across tenants expensive. Directory-based routing allows explicit placement and easier migration, at the cost of a lookup service and another failure domain.
What becomes harder
Transactions across shards need two-phase commit or a saga. Unique constraints are only local unless you use a global service. Joins across shards become application-side merges. Backups, schema migrations, and capacity planning multiply by the number of shards. A shard can become hot even when the hash is uniform, because one tenant or one key dominates traffic.
Resharding
Adding a shard with modulo hashing remaps most keys. Consistent hashing or a directory-based split moves less data, but still requires copying rows, dual-writing, or pausing writes. Plan the migration with a backfill, a verification period, and a rollback path. Test the resharding procedure on a copy before the production run.
When to use it
Use sharding when a single database cannot meet write throughput, storage, or isolation requirements after simpler options are exhausted. The advantage is horizontal capacity; the drawback is that cross-shard operations and operations become application concerns. Compared to vertical scaling or read replicas, sharding is more complex and should be justified by measured limits.