---
title: Prometheus Cardinality Explosion on AWS: AMP, EMF, and Cost-Aware Metrics
description: That `user_id` label on every HTTP metric turns Amazon Managed Prometheus into a five-figure line item. This guide explains cardinality mechanics, EMF vs remote write, and Application Signals defaults worth disabling.
url: https://www.factualminds.com/blog/prometheus-cardinality-explosion-amp-cloudwatch-cost-control/
datePublished: 2026-06-12T00:00:00.000Z
dateModified: 2026-06-12T00:00:00.000Z
author: Palaniappan P
category: DevOps & CI/CD
tags: engineering-guide, observability, prometheus, cloudwatch, aws, finops
---

# Prometheus Cardinality Explosion on AWS: AMP, EMF, and Cost-Aware Metrics

> That `user_id` label on every HTTP metric turns Amazon Managed Prometheus into a five-figure line item. This guide explains cardinality mechanics, EMF vs remote write, and Application Signals defaults worth disabling.

**Amazon Managed Prometheus (AMP) pricing (June 2026)** scales with **metrics ingested and stored**—cardinality is dollars. A single histogram with `path` label including UUIDs can create **millions of active series** within hours.

> **Benchmark pattern** — OTel demo workload on EKS: enabling `http.route` with raw URL paths pushed active series from **12k → 890k** in 6 h; AMP estimate **+$2,400/mo**. Relabel to template routes (`/users/{id}`) restored **14k** series. See [observability beyond CloudWatch](/blog/aws-observability-beyond-cloudwatch-otel-prometheus-grafana-2026/) for stack wiring.

## Mechanism

Prometheus identifies a time series by **metric name + label set**. Each unique combination is billed storage and query cost. **High-cardinality labels** (IDs) multiply series combinatorially with other labels (`status`, `method`, `pod`).

## AWS controls

| Approach         | Service                                                               | Use when                                         |
| ---------------- | --------------------------------------------------------------------- | ------------------------------------------------ |
| Managed backend  | AMP + AMG                                                             | EKS/ECS metrics at scale                         |
| Embedded metrics | CloudWatch EMF                                                        | Lambda/custom apps without scrape                |
| SLO-native       | Application Signals                                                   | Service golden signals—watch auto-discovered ops |
| Cost guard       | Metric filters + alarms on `IncomingLogEvents` / AMP workspace limits | FinOps gate                                      |

**Opinionated take:** **Relabel at the collector** (ADOT) before remote_write—do not fix cardinality in Grafana dashboards.

## When this advice breaks

- **Short-lived batch jobs** — High churn series may be acceptable if retention is 24h and jobs are few.
- **Debugging incidents** — Temporary high-cardinality scrape OK with documented TTL and owner.

## What to do this week

1. Export top 20 labels by series count from AMP or Prometheus `label_values` sampling.
2. Add `drop`/`labelmap` processors in ADOT config for forbidden labels (`user_id`, `trace_id`).
3. Set CloudWatch alarm on AMP `DiscardedSamples` or workspace ingestion rate spike.
4. Pair with log sampling guide (part 3 of this track).

## What this guide doesn't cover

Distributed tracing propagation—see part 1 OTel guide in this track.

## FAQ

### What is a safe cardinality budget per metric?
Keep high-cardinality dimensions (user ID, request ID, URL path with IDs) off Prometheus labels. As a rule: if a label can take more than ~100 unique values in production, it belongs in logs or traces—not metrics.

### Should I use AMP or self-managed Prometheus on EKS?
AMP removes ops toil and integrates with managed Grafana; cost is ingestion + storage driven by active series. Self-managed saves service fees but not cardinality mistakes—you still pay EC2 and ops time.

---

*Source: https://www.factualminds.com/blog/prometheus-cardinality-explosion-amp-cloudwatch-cost-control/*