When the Data Job Runs Out of Memory
Quick summary: A 50 GB or 100 GB CSV loaded whole dies in a small worker. Split the job into interactive SQL, a one-shot conversion, or a repeating pipeline. Adding workers does not split a file nobody partitioned.
Key Takeaways
- A 50 GB or 100 GB CSV loaded whole dies in a small worker
- The failure on 4 October 2026 was an out-of-memory error on a 50 GB CSV, with a 100 GB JSON file in the same bucket waiting to be "just loaded
- The EMR compare chooses a deployment mode
- The data is already on S3 and someone needs an answer, not a cluster
- It is the wrong tool when you are still on a laptop-sized process and the object is 50 GB or 100 GB
Table of Contents
The failure on 4 October 2026 was an out-of-memory error on a 50 GB CSV, with a 100 GB JSON file in the same bucket waiting to be “just loaded.” The reader opened the object whole. The EMR compare chooses a deployment mode. The Glue versus dbt guide chooses a transformation style. Neither starts from this error.
Three shapes
Interactive SQL. The data is already on S3 and someone needs an answer, not a cluster. Athena reads it. The Athena cost post owns how not to scan the lake. Do not load the file into a worker to answer one question.
One-shot conversion. You need the file in a columnar layout once. DuckDB on a single node is the right tool when the object fits that node. It is the wrong tool when you are still on a laptop-sized process and the object is 50 GB or 100 GB. Size the node to the file, or split the file. Do not pretend a laptop success is a design.
A repeating pipeline. Glue or EMR Serverless, after the input is split into objects the workers read independently. The lakehouse pattern owns the platform. The DuckLake versus Iceberg post owns the table-format argument. This page does not.
We recommend splitting the file before adding workers. The trade-off is a conversion step you did not want. More memory on a job that still opens one object moves the crash to a larger machine.
Where this fails
What to do this week
- Find the job that OOMs. Record the object size. Do not record the worker type first.
- If the need is a query, point Athena at the data and stop.
- If the need is one conversion, size a single node or split the file. Then use DuckDB or an equivalent one-shot reader.
- If the need is a pipeline, split first, then Glue or EMR Serverless.
- A lake that keeps producing this file is a data analytics conversation, not another worker type.
What this post doesn’t cover
It does not compare EMR Serverless with EMR on EC2 or EKS, teach Glue versus dbt, or argue DuckLake versus Iceberg. It does not pick a fan-out for a million small items. That comparison is a different article.
Frequently asked questions
Glue, EMR Serverless, Athena, or DuckDB when the file does not fit in memory?
When should we not add Glue workers?
What could go wrong if we use DuckDB because it worked on a laptop sample?
Is this the EMR deployment compare?
Where does Athena cost guidance live?
AWS Cloud Architect & AI Expert
AWS-certified cloud architect and AI expert with deep expertise in cloud migrations, cost optimization, and generative AI on AWS.
