Observability vs Monitoring vs Alerting on AWS: Logs, Metrics, Traces, and the Cost of Noise
Quick summary: CloudWatch Logs Insights charges about $0.005 per GB scanned (US East pricing, May 2026)—a “cheap query” run every minute across full indexes becomes a five-figure monthly line item faster than most teams model.
Key Takeaways
- CloudWatch Logs Insights charges about $0
- 005 per GB scanned (US East pricing, May 2026)—a “cheap query” run every minute across full indexes becomes a five-figure monthly line item faster than most teams model
- The FinOps trap remains Logs Insights: list pricing in public US East documentation centers on $0
- 005 per GB scanned—an innocuous scheduled “wide” query becomes expensive theater when multiplied across teams
- Training wheels: real cost of skipping 24/7 monitoring quantifies the business risk of silent failure
Table of Contents
On May 8, 2026, Amazon CloudWatch is still the default mothership for AWS-native telemetry—but OpenTelemetry with AWS Distro for OpenTelemetry (ADOT) is how teams avoid vendor lock-in while still landing traces in AWS X-Ray. The FinOps trap remains Logs Insights: list pricing in public US East documentation centers on $0.005 per GB scanned—an innocuous scheduled “wide” query becomes expensive theater when multiplied across teams.
This note distinguishes monitoring (continuous observation) from alerting (human routing) and from observability (sufficient context to answer novel questions). Operational mantra from our distributed debugging guide: alert on metrics, triage with traces, diagnose with logs.
Reproduce this — Paste starter queries from
examples/architecture-blog-2026/observability/logs-insights-queries.txtinto Logs Insights; replace log groups and trace IDs.
The three pillars—what each hides
- Metrics aggregate—excellent for SLOs, bad for “why this user.”
- Logs explain—expensive at INFO flood volume without indexing discipline.
- Traces connect—require propagation headers (
traceparent, X-Ray trace IDs) in every hop.
Deep dive patterns live in CloudWatch observability best practices.
Monitoring vs alerting
Monitoring answers “what is normal.” Alerting answers “who acts now.”
Opinionated take — If an alert fires more than twice a week without an action checklist update, delete or re-tier the alarm—noisy paging is a security incident waiting for ignored fatigue.
Training wheels: real cost of skipping 24/7 monitoring quantifies the business risk of silent failure.
OpenTelemetry + learning by chaos
For teams onboarding OTel, OTel Demo Game teaches sampling and failure injection behaviors better than slide decks.
FinOps coupling
Observability spend belongs in the same review as compute—follow the FinOps threads in engineering cost ownership and logging cost deep dive.
What broke — A “temporary” debugging query (
SELECT *across 30 log groups) landed in a CI cron for weeks. Finance flagged a $18k monthly delta—logs were “cheap” until automation scaled humans out of the loop. Fix: query linting, required group filters, and budget alarms per log product.
What This Post Doesn’t Cover
- Third-party APM pricing negotiations—vendor-specific.
- Security analytics in Security Lake vs CloudWatch—different retention/compliance story (Security Lake guide when OCSF normalization matters).
If You Only Do One Thing
Create a SLO dashboard with burn-rate alarms (multi-window) for the three user-visible flows that fund payroll—everything else is secondary tier.
What to Do This Week
- Inventory scheduled Logs Insights queries; delete or constrain time windows.
- Verify W3C trace context crosses service boundaries in staging (not just Lambda console traces).
- Pair every new alarm with a runbook link or delete the alarm—no orphan pages.
When resilience patterns intersect telemetry, read retries, circuits, and graceful shutdown.
AWS Cloud Architect & AI Expert
AWS-certified cloud architect and AI expert with deep expertise in cloud migrations, cost optimization, and generative AI on AWS.
