Bloom Filters and HyperLogLog in Production on ElastiCache Redis
Quick summary: Bloom filters shave 90% of negative lookups; HyperLogLog estimates cardinality without storing every user ID. Redis modules on ElastiCache for abuse detection and feed deduplication.
Key Takeaways
- Bloom filters shave 90% of negative lookups; HyperLogLog estimates cardinality without storing every user ID
- ElastiCache Redis (June 2026) supports Bloom filters (RedisBloom module where enabled) and HyperLogLog ( / ) natively—cheap cardinality for “how many unique IPs today
- HyperLogLog ~12 KB per key for ~0
- 2
- 3
Table of Contents
ElastiCache Redis (June 2026) supports Bloom filters (RedisBloom module where enabled) and HyperLogLog (PFADD/PFCOUNT) natively—cheap cardinality for “how many unique IPs today?” without Prometheus label explosion.
Bloom filter
- Use: “Have we seen this dedup key?” with false positives OK
- Avoid: Need exact membership proof for billing
Configure expected items and false positive rate; size is fixed at creation.
HyperLogLog
~12 KB per key for ~0.81% error on billions of uniques—perfect for dashboard UV metrics, not invoicing.
AWS patterns
| Use case | Structure |
|---|---|
| API abuse | Bloom per IP window in Redis |
| Unique visitors | HLL per day, export to CloudWatch custom metric |
| Feed dedup | Bloom + DynamoDB for positives |
What to do this week
- Replace
SETof all session IDs with HLL for daily active count dashboard. - Add Bloom pre-check before expensive Aurora query.
- Monitor Redis memory—Bloom is cheaper than full SET until false positive rate hurts UX.
What this guide doesn’t cover
General cache invalidation—part 2 of caching track.
AWS Cloud Architect & AI Expert
AWS-certified cloud architect and AI expert with deep expertise in cloud migrations, cost optimization, and generative AI on AWS.
