Kafka on MSK: Partition Rebalancing and Exactly-Once Semantics
Quick summary: Consumer group rebalance storms stall processing longer than broker outages. This guide covers cooperative rebalancing, idempotent producers, and transactional reads on Amazon MSK, and says when SQS FIFO is simpler.
Key Takeaways
- This guide covers cooperative rebalancing, idempotent producers, and transactional reads on Amazon MSK, and says when SQS FIFO is simpler
- Amazon MSK (June 2026) runs managed Kafka brokers with IAM auth and tiered storage options, but consumer group protocol behavior is still Kafka
- A rolling EKS deploy that replaces 30 consumers at once triggers stop-the-world rebalance unless you configure cooperative assignment
- idempotence=true (default since Kafka 3.0) plus transactional
- Benchmark pattern: 24-partition topic, 24 consumers, rolling replace: eager assignor paused processing 38 s; cooperative-sticky <6 s max partition move
Table of Contents
Amazon MSK (June 2026) runs managed Kafka brokers with IAM auth and tiered storage options, but consumer group protocol behavior is still Kafka. A rolling EKS deploy that replaces 30 consumers at once triggers stop-the-world rebalance unless you configure cooperative assignment.
Symptom → mechanism → AWS control
| Production symptom | Mechanism | AWS control |
|---|---|---|
| Duplicate messages on retry | At-least-once without idempotent producer | enable.idempotence=true (default since Kafka 3.0) plus transactional.id |
| Consumer lag spike on deploy | Eager rebalance revokes all partitions | cooperative-sticky assignor, static group membership |
| Transaction timeout aborts | max.transaction.timeout.ms too low | MSK broker config, align with processing SLA |
Opinionated take: Enable idempotent producers and cooperative-sticky rebalancing on MSK before you chase exactly-once. EOS without those two is theater.
Benchmark pattern: 24-partition topic, 24 consumers, rolling replace: eager assignor paused processing 38 s; cooperative-sticky <6 s max partition move. EOS enabled added ~8% producer latency vs idempotent-only. Artifact:
examples/engineering-guides/kafka-msk-partition-rebalancing-exactly-once-semantics/.
Partition rebalancing internals
- Consumer heartbeat missed → coordinator marks member dead.
- Partitions revoked and reassigned (eager = all at once).
- In-flight messages must complete or violate
max.poll.interval.ms.
AWS tip: Run MSK consumers on EKS with PDB (see Kubernetes track) so Kubernetes does not kill half the group simultaneously.
Exactly-once semantics (EOS)
Requires:
enable.idempotence=trueon producertransactional.idunique per producer instance- Consumer
isolation.level=read_committed - Idempotent sink still required. EOS is broker-to-consumer, not into your database without dedupe keys
AWS services map
| Need | Service | Skip when |
|---|---|---|
| Ordered log | MSK | <1k msgs/s, simple queue OK |
| Simpler ordering | SQS FIFO | Need log replay / compaction |
| Stream fan-out | Kinesis | Kafka ecosystem not required |
When this advice breaks
- Cross-Region active-active: MSK MirrorMaker 2 lag; EOS does not span clusters without careful offset mapping.
- RabbitMQ workloads: Different protocol; see Amazon MQ guide in this track.
What to do this week
- Set
partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor. - Enable idempotent producer; add EOS only after sink dedupe proven.
- Alarm on
kafka.consumer.group.rebalancerate and consumer lag p99. - Document rebalance behavior in GameDay runbooks.
More in This Track
Part of the Engineering Guides library (June 2026).
- Next: Part 2
- Browse tracks: Engineering Guides hub
What this guide doesn’t cover
SQS/EventBridge patterns are covered in parts 3 to 5 of this track.
Frequently asked questions
Does MSK support Kafka exactly-once semantics?
Why do MSK consumers pause during deploys?
AWS Cloud Architect & AI Expert
AWS-certified cloud architect and AI expert with deep expertise in cloud migrations, cost optimization, and generative AI on AWS.
Manoj V.