data-engineering

Kinesis Data Streams and Amazon MSK both handle real-time streaming on AWS, but they serve different architectures. Here is how to choose between them for your workload.

Serverless RPUs vs Provisioned RA3 — still not automatic. July 2026 refresh: base 4–1024 RPUs, Max RPU-hours, AI-driven scaling, when RI wins.

Amazon OpenSearch Service powers search, log analytics, and time-series workloads on AWS. Here are the architecture patterns and cost levers that matter most in production.

Kinesis Data Streams combined with Lambda and DynamoDB is the simplest path to a real-time data pipeline on AWS. Here is the complete architecture, code patterns, and operational guidance.

EMR Serverless vs EC2 vs EKS for Spark. July 2026 refresh — Spot limits, cold start, Iceberg on all three, and a decision matrix for intermittent vs 24/7 jobs.

Glue 5.1 (Spark 3.5.6, Iceberg 1.10.0) for lakehouse ETL. July 2026 refresh — Iceberg v2 vs v3, Lake Formation writes, compaction, Athena compatibility.

Glue does EL; dbt does T. July 2026 refresh — Glue 5.1 DPU economics, dbt-athena/Redshift, and the combined stack most teams converge on.

Athena still bills ~$5/TB scanned (or Capacity Reservations by DPU). July 2026 refresh — partitions, Parquet/Iceberg, workgroups, and when provisioned capacity wins.

Managed Flink for stateful streaming. July 2026 refresh — KDA name EOL, KPU + orchestration KPU, per-second billing, Flink vs Lambda decision matrix.