Skip to main content

Blog category

Data & Analytics

Data engineering guides on building AWS data lakes, streaming pipelines with Kinesis, serverless ETL with Glue and Athena, and BI dashboards with QuickSight. Patterns for moving from ad-hoc SQL to production-grade analytics infrastructure.

Articles in this topic
31
A 50 GB CSV opened whole runs out of memory; Athena for questions, DuckDB for one conversion, Glue or EMR Serverless for pipelines, after splitting the input

When the Data Job Runs Out of Memory

A 50 GB or 100 GB CSV loaded whole dies in a small worker. Split the job into interactive SQL, a one-shot conversion, or a repeating pipeline. Adding workers does not split a file nobody partitioned.

2 min read

More in Data & Analytics

4 min

Secure Cross-Account Data Sharing on AWS (2026): Lake Formation, LF-Tags, and Data Mesh Without Copying the Lake

Copying curated Parquet into every consumer account is how data platforms drown in storage cost and permission sprawl. On Feb 11, 2026 AWS shipped Lake Formation cross-account version 5, which adds wildcard RAM shares for hundreds of thousands of tables. A composite 12-account platform cut duplicate curated copies from 3 to 0 and dropped cross-account access tickets from ~11/month to ~3 by standardizing LF-Tags + resource links.

Palaniappan P