---
title: Modern Data Lake on AWS (2026): S3 Tables, Iceberg Compaction, and Analytics Tier Reference Architecture
description: For an enterprise lake (~38 TB Iceberg, 8.4k small files), S3 Tables managed compaction cut Athena scan spend −35% — compaction OPEX −90% vs pre–July 2025 pricing on the same workload.
url: https://www.factualminds.com/blog/aws-modern-data-lake-s3-tables-iceberg-reference-architecture-2026/
datePublished: 2026-07-07T00:00:00.000Z
dateModified: 2026-07-07T00:00:00.000Z
author: palaniappan-p
category: Cloud Architecture
tags: aws, data-lake, s3-tables, iceberg, glue, athena, analytics, architecture
---

# Modern Data Lake on AWS (2026): S3 Tables, Iceberg Compaction, and Analytics Tier Reference Architecture

> For an enterprise lake (~38 TB Iceberg, 8.4k small files), S3 Tables managed compaction cut Athena scan spend −35% — compaction OPEX −90% vs pre–July 2025 pricing on the same workload.

**Amazon S3 Tables** reduced compaction costs up to **90%** effective **July 1, 2025**, with default **512 MB** target files and optional **sort** and **z-order** strategies ([What's New](https://aws.amazon.com/about-aws/whats-new/2025/07/amazon-s3-tables-reduce-compaction-costs/), [sort/z-order GA](https://aws.amazon.com/about-aws/whats-new/2025/06/amazon-s3-sort-z-order-compaction-apache-iceberg-tables/)). **AWS Glue 5.1** ships **Spark 3.5.6**, **Iceberg 1.10+**, and **Lake Formation fine-grained write** controls ([Glue 5.1 GA](https://aws.amazon.com/about-aws/whats-new/2025/11/aws-glue-5-1/)).

This post is the **horizontal modern data lake reference architecture** — ingest, curated Iceberg, catalog, consumption. It is **not** the [legacy S3+Glue primer](/blog/building-a-data-lake-on-aws-s3-glue-athena-architecture/), **not** [governance operating model](/blog/aws-data-governance-operating-model-sagemaker-catalog-2026/), and **not** [retail omnichannel lakehouse](/blog/aws-retail-omnichannel-analytics-lakehouse-2026/).

Artifacts: [lake tier matrix](https://www.factualminds.com/examples/architecture-blog-2026/modern-data-lake/lake-tier-decision-matrix.md), [compaction cost worksheet](https://www.factualminds.com/examples/architecture-blog-2026/modern-data-lake/iceberg-compaction-cost-worksheet.csv), [ingest partition checklist](https://www.factualminds.com/examples/architecture-blog-2026/modern-data-lake/ingest-partition-checklist.md).

> **Benchmark pattern (not a cited client)** — **Enterprise analytics lake**, **38 TB** Iceberg curated, **8,400** files averaging **12 MB**, promotion-week queries. **S3 Tables binpack compaction** (**Athena scan spend −35%** modeled). **Compaction OPEX −90%** vs pre–Jul 2025 pricing ([worksheet](https://www.factualminds.com/examples/architecture-blog-2026/modern-data-lake/iceberg-compaction-cost-worksheet.csv)). **Glue 5.1** ETL with LF write policies.

## Four tiers — raw to curated to consume

| Tier        | Question         | Default AWS path                                    |
| ----------- | ---------------- | --------------------------------------------------- |
| **Landing** | Immutable raw?   | S3 `raw/` + Object Lock if required                 |
| **Curated** | ACID tables?     | **S3 Tables** Iceberg + managed maintenance         |
| **Catalog** | Discovery + ETL? | **Glue 5.1** Data Catalog                           |
| **Consume** | Who queries?     | **Athena** + **Redshift Spectrum** / **QuickSight** |

**Opinionated take:** **Turn on S3 Tables maintenance before hiring a compaction squad.** Teams that manually compact 8k-file tables every weekend burn platform budget that July 2025 pricing was meant to eliminate.

## Reference architecture

```
Sources ──► Glue 5.1 / Firehose ──► S3 raw ──► Glue Spark job ──► S3 Tables (Iceberg)
                                                      │
                                              Glue Data Catalog
                                                      │
                                    ┌─────────────────┴─────────────────┐
                                    ▼                                   ▼
                                 Athena                            Redshift Spectrum
                                    │
                              QuickSight / SageMaker
Lake Formation ──► column/table tags ──► FGAC on read + write (Glue 5.1)
```

### Tier 1 — Landing

Follow [ingest-partition-checklist.md](https://www.factualminds.com/examples/architecture-blog-2026/modern-data-lake/ingest-partition-checklist.md):

- Partition on `day(event_ts)` — not high-cardinality IDs
- Schema evolution tracked in Glue

### Tier 2 — S3 Tables maintenance

Per [S3 Tables maintenance docs](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-maintenance.html):

- Default compaction **512 MB** target (64–512 configurable)
- Strategies: `auto`, `binpack`, `sort`, `z-order`
- Snapshot expiry: default **MaximumSnapshotAge 120 hours** — align with legal hold

```bash
# AWS CLI — set sort compaction when table sort order defined (Glue 5.1 + S3 Tables)
aws s3tables put-table-maintenance-configuration \
  --table-bucket-arn arn:aws:s3tables:us-east-1:111122223333:bucket/amzn-s3-demo-bucket \
  --type icebergCompaction \
  --namespace analytics \
  --name events_curated \
  --value='{"status":"enabled","settings":{"icebergCompaction":{"strategy":"sort","targetFileSizeMB":256}}}'
```

### Tier 3 — Glue 5.1 catalog + ETL

- Spark **3.5.6**, **Iceberg format v3** features where needed
- **Lake Formation** write FGAC on INSERT/overwrite
- Link stewardship to [governance operating model](/blog/aws-data-governance-operating-model-sagemaker-catalog-2026/)

### Tier 4 — Consumption

- Athena engine v3 on Iceberg — verify partition pruning in query plans
- Redshift: see [Iceberg best practices](https://aws.amazon.com/blogs/big-data/best-practices-for-querying-apache-iceberg-data-with-amazon-redshift/)

> **What broke** — Q2 promotion post-mortem. **Same dashboard query 3.1× Athena cost** week-over-week. Root cause: daily micro-batches left **8,400× 12 MB** files — planner scanned all manifests. **Fix:** S3 Tables compaction enabled; **scan −35%** on replay ([worksheet](https://www.factualminds.com/examples/architecture-blog-2026/modern-data-lake/iceberg-compaction-cost-worksheet.csv)).

## When NOT to escalate

| Situation                   | Stay lighter                        |
| --------------------------- | ----------------------------------- |
| < 500 GB total           | Hive external tables + weekly batch |
| Single team, no ACID merges | Parquet on S3 may suffice           |
| No LF admin                 | Fix IAM sprawl before FGAC project  |

## What to Do This Week

1. Run [lake-tier-decision-matrix.md](https://www.factualminds.com/examples/architecture-blog-2026/modern-data-lake/lake-tier-decision-matrix.md).
2. Audit file count + average size — if > 1,000 files < 32 MB, enable compaction.
3. Reprice [iceberg-compaction-cost-worksheet.csv](https://www.factualminds.com/examples/architecture-blog-2026/modern-data-lake/iceberg-compaction-cost-worksheet.csv).
4. Upgrade Glue jobs to **5.1** — test LF write on staging table.
5. Complete [ingest-partition-checklist.md](https://www.factualminds.com/examples/architecture-blog-2026/modern-data-lake/ingest-partition-checklist.md) before next major ingest.

> **Reproduce this** — Download [iceberg-compaction-cost-worksheet.csv](https://www.factualminds.com/examples/architecture-blog-2026/modern-data-lake/iceberg-compaction-cost-worksheet.csv). Plug in `small_files_count` from `aws s3 ls` sample or Iceberg metadata table. Compare `monthly_compaction_usd_after_jul2025` to self-managed Spark row.

## What This Post Doesn't Cover

- **Real-time streaming lake** — Kinesis/Flink path (see event throughput post)
- **DataZone business catalog UX** — governance post covers stewardship
- **Cross-account Lake Formation shares** — [sharing guide](/blog/aws-secure-cross-account-data-sharing-lake-formation-2026/)
- **Mainframe CDC** — partner tooling

We have not tested **z-order compaction** on tables without declared sort order — prerequisites per AWS docs require sort metadata before enabling `sort`/`z-order`.

**Related:** [Data analytics services](/services/aws-data-analytics/) · [SageMaker services](/services/aws-sagemaker/) · [Data governance post](/blog/aws-data-governance-operating-model-sagemaker-catalog-2026/)

## FAQ

### When should we adopt S3 Tables vs self-managed Iceberg on S3?
S3 Tables when you want managed compaction, snapshot expiry, and unreferenced file removal without a platform SRE team. Self-manage only when you need custom compaction schedules Glue cannot express or regulatory hold on snapshot deletion policies.

### When should we NOT move to Iceberg yet?
When consumers are CSV-only and full scans cost <$5 — external Hive tables may suffice until join complexity or ACID merges appear.

### Glue 5.0 vs 5.1 for new lakes?
Default 5.1 (Spark 3.5.6, Iceberg 1.10+, Iceberg v3 features, LF write FGAC). 5.0 only if your region lacked 5.1 at cutover — check April 2026 GA notice.

### How does this differ from the data governance operating model post?
Governance post covers stewardship RACI and SageMaker Catalog on DataZone. This post is the storage + maintenance + query tier wiring for Iceberg lakes.

### What breaks with too many small Iceberg files?
Athena planning time and scan cost explode — symptom: same query 3× cost after bad daily loads. Fix: enable compaction or lower ingest batch size.

### Does retail lakehouse replace this pattern?
No — retail post is CDP/omnichannel event shapes. This is horizontal enterprise lake tiers any industry can reuse.

---

*Source: https://www.factualminds.com/blog/aws-modern-data-lake-s3-tables-iceberg-reference-architecture-2026/*