Prometheus Cardinality Explosion on AWS: AMP, EMF, and Cost-Aware Metrics
Quick summary: That `user_id` label on every HTTP metric turns Amazon Managed Prometheus into a five-figure line item. This guide explains cardinality mechanics, EMF vs remote write, and Application Signals defaults worth disabling.
Key Takeaways
- That `user_id` label on every HTTP metric turns Amazon Managed Prometheus into a five-figure line item
- Amazon Managed Prometheus (AMP) pricing (June 2026) scales with metrics ingested and stored — cardinality is dollars
- Since July 31, 2026, there is a second Prometheus path on AWS: CloudWatch managed collectors can scrape the same exporters into a CloudWatch OpenTelemetry dataset at $0
- 50/GB ingested (15 months storage, no per-series custom-metric fee) instead of AMP samples — if your pane is CloudWatch and you fix labels before scrape
- For AWS Example 22 shape (100 hosts × 50 metrics @ 60s), OTLP is $54/mo vs $1,500/mo classic custom metrics — see the collectors post cost CSV
Table of Contents
Amazon Managed Prometheus (AMP) pricing (June 2026) scales with metrics ingested and stored — cardinality is dollars. A single histogram with path label including UUIDs can create millions of active series within hours.
Since July 31, 2026, there is a second Prometheus path on AWS: CloudWatch managed collectors can scrape the same /metrics exporters into a CloudWatch OpenTelemetry dataset at $0.50/GB ingested (15 months storage, no per-series custom-metric fee) instead of AMP samples — if your pane is CloudWatch and you fix labels before scrape. Cardinality still multiplies OTLP GB; the billing meter changes, not the hygiene. For AWS Example 22 shape (100 hosts × 50 metrics @ 60s), OTLP is $54/mo vs $1,500/mo classic custom metrics — see the collectors post cost CSV. AMP remains the right backend when you need recording rules and Grafana-native workflows.
Benchmark pattern — OTel demo workload on EKS: enabling
http.routewith raw URL paths pushed active series from 12k → 890k in 6 h; AMP estimate +$2,400/mo. Relabel to template routes (/users/{id}) restored 14k series. See observability beyond CloudWatch for stack wiring.
Symptom → mechanism → AWS control
| Production symptom | Mechanism | AWS control |
|---|---|---|
| Observability bill exceeds compute | High-cardinality labels (user_id, pod) | AMP relabel_configs drop, aggregate recording rules |
| CloudWatch PutMetricData throttled | Custom metric cardinality limits | EMF embedded metrics with bounded dimensions |
| Dashboard slow to load | Million-series PromQL scan | AMP query logging, cardinality governance policy |
Opinionated take: Treat metric labels like database indexes—approve new high-cardinality labels in review, and drop pod-name from application metrics at scrape time.
Mechanism
Prometheus identifies a time series by metric name + label set. Each unique combination is billed storage and query cost. High-cardinality labels (IDs) multiply series combinatorially with other labels (status, method, pod).
AWS controls
| Approach | Service | Use when |
|---|---|---|
| Managed scrape | CloudWatch managed collectors → OTLP (Jul 2026) | CloudWatch-first; metrics-only scrape fleet today |
| Managed backend | AMP + AMG | EKS/ECS metrics at scale; Grafana/recording rules |
| Embedded metrics | CloudWatch EMF | Lambda/custom apps without scrape |
| SLO-native | Application Signals | Service golden signals—watch auto-discovered ops |
| Cost guard | Metric filters + alarms on IncomingLogEvents / AMP workspace limits | FinOps gate |
Opinionated take: Relabel at scrape time — managed collectors, ADOT, or Prometheus metric_relabel_configs — before remote_write or OTLP export. Do not fix cardinality in Grafana dashboards.
For destination choice (CloudWatch OTLP vs AMP vs classic custom metrics), see CloudWatch managed Prometheus collectors (July 2026).
AWS services map
| Need | Service | Skip when |
|---|---|---|
| Cardinality control | AMP + relabel/aggregate rules | CloudWatch only with <10 custom metrics |
| Cost attribution | Cost and Usage Report + AMP usage metrics | Single-service app with default metrics |
| FinOps guardrails | CloudWatch anomaly detection on AMP spend | Pre-production with negligible ingest |
When this advice breaks
- Short-lived batch jobs — High churn series may be acceptable if retention is 24h and jobs are few.
- Debugging incidents — Temporary high-cardinality scrape OK with documented TTL and owner.
What to do this week
- Export top 20 labels by series count from AMP or Prometheus
label_valuessampling. - Add
drop/labelmapprocessors in ADOT config for forbidden labels (user_id,trace_id). - Set CloudWatch alarm on AMP
DiscardedSamples, OTel ingestion GB spike, or workspace ingestion rate spike. - If CloudWatch is your pane and you scrape
/metricstoday, compare AMP samples vs managed collector → OTLP with the published cost CSV before the next scrape fleet expansion. - Pair with log sampling guide (part 3 of this track).
More in This Track
Part of the Engineering Guides library (June 2026).
- Previous: Part 1
- Next: Part 3
- Browse tracks: Engineering Guides hub
What this guide doesn’t cover
Distributed tracing propagation—see part 1 OTel guide in this track.
AWS Cloud Architect & AI Expert
AWS-certified cloud architect and AI expert with deep expertise in cloud migrations, cost optimization, and generative AI on AWS.
