---
title: DuckLake vs Apache Iceberg on AWS (2026): SQL Catalog or S3 Tables
description: DuckLake spec 1 shipped on 13 April 2026 and keeps lake metadata in a SQL database. DuckDB's architecture note puts Iceberg planning at about 100 ms per sequential S3 GET, roughly half a second before a query starts. On AWS, stay on S3 Tables when Athena, Glue, or Redshift must read the table.
url: https://www.factualminds.com/blog/ducklake-vs-apache-iceberg-on-aws-2026/
datePublished: 2026-09-22T00:00:00.000Z
dateModified: 2026-09-22T00:00:00.000Z
author: palaniappan-p
category: Cloud Architecture
tags: ducklake, iceberg, duckdb, s3, aurora, athena, s3-tables
---

# DuckLake vs Apache Iceberg on AWS (2026): SQL Catalog or S3 Tables

> DuckLake spec 1 shipped on 13 April 2026 and keeps lake metadata in a SQL database. DuckDB's architecture note puts Iceberg planning at about 100 ms per sequential S3 GET, roughly half a second before a query starts. On AWS, stay on S3 Tables when Athena, Glue, or Redshift must read the table.

Choose DuckLake only when every reader and writer is DuckDB and the catalog runs on Aurora PostgreSQL or RDS for PostgreSQL, with Parquet on S3.

Teams that already operate an Iceberg lake should start from the [S3 Tables reference architecture](/blog/aws-modern-data-lake-s3-tables-iceberg-reference-architecture-2026/). That post covers compaction pricing. This one is the format choice.

**DuckLake spec 1** shipped on **13 April 2026**, with the reference `ducklake` extension in **DuckDB 1.5.2**. The specification is frozen with backward compatibility. The [DuckLake release calendar](https://ducklake.select/release_calendar.html) still lists spec 1.1 as tentative for fall 2026. This guide is written against spec 1. DuckDB 1.5.0 and 1.5.1 speak spec 0.4. Do not attach a spec 1 catalog with those builds.

On AWS, keep **Apache Iceberg on Amazon S3 Tables** when Athena, Glue, Redshift, or Spark must read the table.

The trade-off is planning latency versus engine coverage. In a [MotherDuck architecture walkthrough](https://motherduck.com/videos/ducklake-architecture-deep-dive/) of the DuckDB design, each sequential object-storage GET is about **100 ms**. Iceberg reads a metadata file, then a manifest list, then a manifest, so planning sits at about **half a second** before the query touches Parquet. DuckLake puts that metadata in the SQL catalog, so planning is one query. FactualMinds has not measured that half-second. It is DuckDB's stated figure.

Artifacts: [decision worksheet](https://www.factualminds.com/examples/architecture-blog-2026/ducklake-vs-iceberg/decision-worksheet.csv), [attach SQL](https://www.factualminds.com/examples/architecture-blog-2026/ducklake-vs-iceberg/attach-ducklake-s3.sql).

> **Published figure** — MotherDuck's DuckLake architecture deep dive, restating the DuckDB design: about **100 ms** per object-storage GET, and about **0.5 s** of sequential metadata reads before an Iceberg query starts. Not a FactualMinds benchmark. Not a client result.

## What DuckLake is

DuckLake is an open lakehouse format from DuckDB Labs, MIT licensed. Data files are Parquet on object storage. **All metadata lives in a SQL database** that supports ACID transactions and primary keys. The [v1.0 specification](https://ducklake.select/docs/stable/specification/introduction) defines that catalog as **28 tables**, including snapshots, file paths, column statistics, and the rows staged by data inlining.

The `ducklake` extension is the reference implementation. Official catalog databases for that extension are **PostgreSQL, SQLite, and DuckDB**. A single DuckDB file is enough on a laptop. The [DuckLake FAQ](https://ducklake.select/) names **Amazon Aurora** as a production catalog host, with the Parquet prefix on S3.

There is no separate catalog HTTP API. A commit is a SQL transaction: file path, stats, snapshot id, and the change log land together. Cross-table changes share that transaction. The May 2025 manifesto states the design target as a PostgreSQL catalog at hundreds of terabytes and thousands of compute nodes. That sentence is DuckDB's claim. This post does not treat it as a measured limit.

## What Apache Iceberg is

Apache Iceberg is the open table format AWS analytics engines already speak. Data is Parquet (or another Iceberg data format) on S3. Table state is a tree of files: a metadata JSON file, a manifest list, and manifest files that name the data files in a snapshot. A catalog — **Glue Data Catalog** or **S3 Tables** — stores the pointer to the current metadata file so readers agree on the latest snapshot.

That split is why Iceberg works across engines. Athena, Redshift Spectrum, Glue Spark, EMR, and DuckDB's `iceberg` extension can all plan a query once they can read the catalog pointer and the metadata files. It is also why planning touches object storage several times before a single data file is opened.

On AWS, the maintained path is **S3 Tables** plus **Glue 5.1**. S3 Tables reduced compaction costs by up to **90%** effective **1 July 2025**, with a default **512 MB** target file ([AWS What's New](https://aws.amazon.com/about-aws/whats-new/2025/07/amazon-s3-tables-reduce-compaction-costs/)). Glue 5.1 ships Spark 3.5.6 and Iceberg 1.10+ ([Glue 5.1 GA, November 2025](https://aws.amazon.com/about-aws/whats-new/2025/11/aws-glue-5-1/)). Scan-cost tactics for those tables are in the [Athena Iceberg cost guide](/blog/athena-query-cost-optimization-partition-compress-cache-iceberg/).

## Why use DuckLake

Use it when the pain is the metadata path and the small-write path, and DuckDB is the only engine.

- **One round trip to plan.** The catalog database returns the file list. You skip the metadata-file → manifest-list → manifest chain. Who this is for: interactive DuckDB queries where half a second of planning dominates the scan.
- **Small changes stay out of S3 until you flush them.** v1.0 turns inlining on by default at **10 rows**. An insert, update, and delete under that threshold write no Parquet file until `CHECKPOINT`. Who this is for: streams of row-sized changes that would otherwise create a tiny file per commit.
- **Several tables commit together.** Schema changes and data changes are rows in the same database transaction. Who this is for: a load that must not publish table A without table B.
- **No new catalog service.** If Aurora or RDS PostgreSQL is already operated, DuckLake adds tables to a database you know how to back up. Who this is for: a platform team that will not run a second catalog product.

**Opinionated take:** take DuckLake for a DuckDB-only lake with a Postgres catalog you already run. Do not take it to escape Glue while Athena is still a consumer. The half-second planning claim is real as a DuckDB design argument. Losing Athena is the price.

## Why use Iceberg on AWS

Use it when more than one engine must read the same table, or when you want AWS to run maintenance.

- **Athena, Redshift, Glue, and EMR read it.** A table QuickSight hits through Athena is an Iceberg table. DuckLake is invisible to those services.
- **S3 Tables runs compaction, snapshot expiry, and unreferenced-file removal.** The July 2025 price cut is the reason to turn maintenance on before staffing a weekend compaction job. The worksheet in the [modern data lake post](/blog/aws-modern-data-lake-s3-tables-iceberg-reference-architecture-2026/) is the cost model for that choice.
- **Lake Formation is the grant plane.** Column and table grants, cross-account shares, and Glue as the system of record do not apply to a DuckLake catalog in Aurora.
- **The metadata survives without your OLTP database.** If Glue and S3 are up, Athena can still plan. DuckLake cannot plan when Postgres is down.

## When to use which

Match the first row that fits. The same rules are in the [worksheet](https://www.factualminds.com/examples/architecture-blog-2026/ducklake-vs-iceberg/decision-worksheet.csv).

| Engines that must read | Writers | Catalog you already run | Choice |
| --- | --- | --- | --- |
| Athena, Redshift, Glue Spark, EMR, or QuickSight | Any | Any | **Iceberg on S3 Tables** |
| DuckDB only | Two or more | A single DuckDB file | **Stop.** Move the catalog to Aurora or RDS PostgreSQL before the second writer |
| DuckDB only | One or more | Aurora PostgreSQL or RDS PostgreSQL | **DuckLake**, `DATA_PATH` on S3 |
| DuckDB only | One | Nothing yet | Local `.ducklake` file for a prototype, then Aurora before anyone else writes |
| Athena or QuickSight still on the table | Any | Glue | **Stay on Iceberg.** DuckLake is not a Glue exit |

A batch-append DuckDB lake can live in either format. If an AWS analytics engine is on the roadmap for the next two quarters, start on Iceberg. Migrating the catalog later is a project. Picking Iceberg first is a catalog pointer.

## How this works on AWS

Two stacks. They share S3. They do not share a catalog.

```mermaid
flowchart LR
  duckdb[DuckDB_1_5_2]
  aurora[Aurora_PostgreSQL_catalog]
  s3[S3_Parquet]
  athena[Athena_Glue_Redshift]
  s3tables[S3_Tables_Iceberg]
  glue[Glue_Data_Catalog]
  duckdb --> aurora
  duckdb --> s3
  athena --> s3tables
  athena --> glue
```

### DuckLake path

1. **Catalog.** Aurora PostgreSQL or RDS for PostgreSQL, Multi-AZ if more than one writer depends on it. The extension creates the 28 catalog tables. Network path: the DuckDB host must reach port 5432. A laptop on a bastion or SSM port forward is enough for a prototype. Shared writers belong on EC2 or ECS in the same VPC, with the Aurora security group limited to that fleet.
2. **Storage.** An S3 prefix, for example `s3://bucket/ducklake/`. Files are immutable. The extension does not rewrite a key in place. A gateway endpoint keeps that traffic off the NAT gateway.
3. **Compute.** Any host running **DuckDB 1.5.2 or a later 1.5.x** build with the `ducklake`, `postgres`, and `httpfs` extensions. Many DuckDB processes can attach the same Postgres catalog. That is the multi-writer model. A single-file DuckDB catalog is not.
4. **Auth.** Postgres roles protect the catalog. An IAM principal protects the prefix. `CREATE SECRET (TYPE s3, PROVIDER credential_chain)` uses the instance role, environment, or the local AWS profile. DuckLake does not issue Lake Formation credentials.
5. **What AWS will not query.** Athena, Glue Data Catalog, Redshift Spectrum, EMR Spark, and Lake Formation do not read DuckLake. Community clients exist — Apache DataFusion, a MotherDuck Spark client, and Trino ports named in the [v1.0 announcement](https://ducklake.select/2026/04/13/ducklake-10/). This guide does not treat those as AWS-native readers.

MotherDuck's hosted catalog is a different product. It is not this architecture.

### Iceberg path

Writers are Glue 5.1 jobs or Athena `MERGE`. Storage is an S3 table bucket with managed maintenance. The catalog pointer lives in S3 Tables and, when you want one directory for the account, in the Glue Data Catalog via SageMaker Lakehouse. Readers are Athena, Redshift, and QuickSight. Wiring, compaction settings, and the cost worksheet are in the [S3 Tables reference architecture](/blog/aws-modern-data-lake-s3-tables-iceberg-reference-architecture-2026/). Do not rebuild that lake here.

DuckDB can **read** those Iceberg tables. Current DuckDB docs still mark S3 Tables support as **experimental**.

## Attach DuckLake to Aurora and S3

DuckDB 1.5.2 or later, `ducklake` extension 1.0, spec 1.0. Replace the host and the bucket. Do not commit a password. The same script is [attach-ducklake-s3.sql](https://www.factualminds.com/examples/architecture-blog-2026/ducklake-vs-iceberg/attach-ducklake-s3.sql).

```sql
INSTALL ducklake;
INSTALL postgres;
INSTALL httpfs;
LOAD ducklake;
LOAD postgres;
LOAD httpfs;

CREATE SECRET (
  TYPE s3,
  PROVIDER credential_chain,
  REGION 'us-east-1'
);

ATTACH 'ducklake:postgres:dbname=ducklake_catalog host=catalog.cluster-example.us-east-1.rds.amazonaws.com'
  AS lake (DATA_PATH 's3://example-bucket/ducklake/');

USE lake;

CREATE TABLE IF NOT EXISTS events (id BIGINT, status VARCHAR, ts TIMESTAMP);
INSERT INTO events VALUES (1, 'en route', TIMESTAMP '2026-04-13 00:00:00');
```

The connection string matches the [DuckDB DuckLake usage docs](https://blobs.duckdb.org/docs/ducklake-docs.md): `ducklake:postgres:` plus a libpq string, and `DATA_PATH` set to the S3 prefix. One inserted row is under the **10-row** inlining default, so `ducklake_list_files('lake', 'events')` returns no Parquet file until `CHECKPOINT`.

## Read Iceberg on S3 Tables from DuckDB

DuckDB current docs, `iceberg` extension, S3 Tables support marked experimental. This attaches a table bucket. It does not create the lake. Production writes still belong in Glue or Athena, as in the reference architecture post.

```sql
INSTALL aws;
INSTALL httpfs;
INSTALL iceberg;

CREATE SECRET (
  TYPE s3,
  PROVIDER credential_chain
);

ATTACH 'arn:aws:s3tables:us-east-1:111122223333:bucket/example'
  AS ice (TYPE iceberg, ENDPOINT_TYPE s3_tables);

SHOW ALL TABLES;
```

`ENDPOINT_TYPE s3_tables` sets SigV4 and the regional S3 Tables REST endpoint. It cannot be combined with a hand-written `ENDPOINT`. Account `111122223333` in the ARN is the [AWS documentation sample account](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference-arns.html), not a FactualMinds account.

## What broke

> **What broke** — Documented in the DuckDB DuckLake usage page, not observed in a FactualMinds engagement. `ATTACH … (DATA_PATH 'other/', OVERRIDE_DATA_PATH true)` does **not** update the path stored in the catalog. It hides files under the original `DATA_PATH` for that connection. Queries return no rows for data that is still on S3. Detection: `ducklake_list_files` names keys you cannot read, and a second session without the override still sees them. Recovery: detach, attach without `OVERRIDE_DATA_PATH`, or point `DATA_PATH` at the prefix recorded in the catalog.

A second failure is a design consequence, not an incident report. If Aurora is unavailable, every DuckDB session stops planning, including readers. The Parquet objects are still in the bucket. Iceberg on S3 Tables does not have that dependency: Athena plans from Glue and the metadata files. Run the DuckLake catalog Multi-AZ, snapshot it, and alarm on Postgres availability the same way you alarm on a warehouse. A single-AZ dev instance is fine for the prototype in the SQL file. It is the wrong catalog once a second writer exists.

## What to Do This Week

1. List every engine that must read the table this quarter. Include QuickSight. If Athena, Redshift, Glue Spark, or EMR appears, stay on S3 Tables and stop.
2. If the list is DuckDB only, open the [worksheet](https://www.factualminds.com/examples/architecture-blog-2026/ducklake-vs-iceberg/decision-worksheet.csv) and fill `your_row`. Match it to a recommendation row. Do not average the rows.
3. If the match is DuckLake, run [attach-ducklake-s3.sql](https://www.factualminds.com/examples/architecture-blog-2026/ducklake-vs-iceberg/attach-ducklake-s3.sql) against an **empty** prefix and a dev Aurora database. Run `SELECT version();` first and stop if it is older than 1.5.2. Insert one row, call `ducklake_list_files`, then `CHECKPOINT`, and confirm a Parquet object appears under the prefix.

> **Reproduce this** — Download [decision-worksheet.csv](https://www.factualminds.com/examples/architecture-blog-2026/ducklake-vs-iceberg/decision-worksheet.csv). Fill the `your_row` engines, writer count, catalog, and write shape. Read the `recommendation` on the first matching rule row. The SQL file is the attach step when that recommendation is `ducklake-on-aurora-and-s3` or `ducklake-local-file-catalog`.

## What This Post Doesn't Cover

- No FactualMinds benchmark of planning time, inlining, or Aurora catalog load. The 100 ms and 0.5 s figures are DuckDB's, via the MotherDuck walkthrough.
- MotherDuck's hosted DuckLake, where they run the catalog and the storage. The AWS path in this post is your Aurora database and your S3 prefix.
- A claim that registering existing Parquet, or DuckLake's Iceberg-compatible deletion vectors, makes the table readable by Athena. Deletion vectors in v1.0 are experimental in DuckDB's own announcement. Athena still needs an Iceberg catalog.
- DuckLake **1.1**, still unshipped as of 22 September 2026. The v1.0 post schedules variant inlining for non-DuckDB catalogs, and multi-deletion-vector Puffin files, for 1.1. Roles, git-style branching, and incremental materialized views are on the **v2.0** list, not 1.1.
- DataFusion, Spark, and Trino clients. Named in the v1.0 announcement. Not tested for this post. Not a substitute for Athena.

## If You Only Do One Thing

Write down the engines. If any AWS analytics engine is on the list, the format is Iceberg on S3 Tables. DuckLake is the lake you run when that list is DuckDB and the catalog is Postgres you already operate.

## FAQ

### When should we not leave Apache Iceberg for DuckLake?
Stay on Iceberg when Athena, Amazon Redshift, AWS Glue Spark, Amazon EMR, or QuickSight must query the table. Those engines read Iceberg. They do not read DuckLake. Also stay if Lake Formation is the access-control plane, or if the team will not operate a PostgreSQL catalog.

### What goes wrong if the PostgreSQL catalog is unavailable?
DuckLake planning stops. Parquet files can still sit on S3, but a query cannot learn which files belong to the current snapshot without the catalog database. Iceberg on S3 Tables can still be planned from Glue and the metadata files when the query engine is Athena. Treat Aurora Multi-AZ as part of the lake, not as an optional cache.

### Does Amazon Athena read DuckLake tables?
No. Athena reads Apache Iceberg, including tables in Amazon S3 Tables. DuckLake is a different format. Registering existing Parquet into a DuckLake catalog does not make that table an Athena table.

### What does the 10-row inlining default actually do?
DuckLake v1.0 turns data inlining on by default with a threshold of 10 rows. Inserts, updates, and deletes at or under that size stay in the catalog database until CHECKPOINT writes Parquet. ducklake_list_files can return no files before that checkpoint. Raise or zero the limit with data_inlining_row_limit when you want every commit to land a file.

### We already pay for Glue. Should we still consider DuckLake?
Only if every consumer of the new tables is DuckDB. Glue, S3 Tables maintenance, and Lake Formation are the reason to keep Iceberg for shared analytics. S3 Tables cut compaction charges by up to 90 percent from 1 July 2025. DuckLake does not replace that maintenance plane for Athena users.

### How do authentication and encryption work?
DuckLake uses the catalog database for authentication. On Aurora PostgreSQL that is Postgres roles, network controls, and whatever auth you already run. Parquet on S3 still needs an IAM permission to the DATA_PATH prefix. The extension also accepts ENCRYPTED on ATTACH. Postgres roles are not Lake Formation column grants.

---

*Source: https://www.factualminds.com/blog/ducklake-vs-apache-iceberg-on-aws-2026/*