HyperLogLog Cardinality
HyperLogLog estimates the number of distinct values in a stream using fixed memory, trading exactness for the ability to count uniques that exceed memory in a set.
Definition
HyperLogLog is a probabilistic algorithm that estimates the cardinality (number of distinct values) of a multiset using a fixed, configurable amount of memory by hashing values and tracking the maximum leading-zero run length, producing an estimate with known error bounds at a fraction of the memory of an exact set.
Context and scope
Applied in analytics (unique visitors, distinct queries), database cardinality estimation, and any streaming count-distinct where memory is bounded.
Common misconceptions
- HyperLogLog is exact for small counts — it is an estimate at all counts; for small counts it can be corrected, but it is never exact
- A larger register always means a better estimate — it means lower relative error; the estimate is still probabilistic and can be off in either direction
Anti-patterns
- Using HyperLogLog where an exact count is required (financial reconciliation) and treating the estimate as truth
- Merging HyperLogLog sketches from different hash functions, producing a meaningless combined estimate
Related concepts
Authoritative source
What it is
HyperLogLog is a probabilistic algorithm that estimates the cardinality (number of distinct values) of a multiset using a fixed, configurable amount of memory by hashing values and tracking the maximum leading-zero run length, producing an estimate with known error bounds at a fraction of the memory of an exact set. Applied in analytics (unique visitors, distinct queries), database cardinality estimation, and any streaming count-distinct where memory is bounded. The data model and the access patterns are co-designed: a schema that is correct in the relational sense can be catastrophic for the queries the workload actually issues.
How it works
Indexing decisions follow the query shapes, not the column importance: a covering index turns a table scan into an index-only scan, but every index adds write amplification and storage cost. Partitioning aligns with the access dimension so a query touches one partition, not all. Transaction scope is minimized to hold locks briefly; long transactions are the root of contention and deadlock.
When to apply and pitfalls
Apply when data volume or query latency has a concrete business cost. The common pitfall is indexing for the current workload and ignoring the access patterns that growth will introduce — the index that helps today can serialize writes tomorrow. Another is treating the database as a black box, missing that the optimizer’s plan choice depends on statistics that drift as data changes.