--- title: When the Data Job Runs Out of Memory description: A 50 GB or 100 GB CSV loaded whole dies in a small worker. Split the job into interactive SQL, a one-shot conversion, or a repeating pipeline. Adding workers does not split a file nobody partitioned. url: https://www.factualminds.com/blog/large-file-processing-glue-emr-athena-duckdb/ datePublished: 2026-11-10T00:00:00.000Z dateModified: 2026-11-10T00:00:00.000Z author: palaniappan-p category: Data & Analytics tags: aws, aws-glue, amazon-athena, amazon-emr --- # When the Data Job Runs Out of Memory > A 50 GB or 100 GB CSV loaded whole dies in a small worker. Split the job into interactive SQL, a one-shot conversion, or a repeating pipeline. Adding workers does not split a file nobody partitioned. The failure on 4 October 2026 was an out-of-memory error on a **50 GB** CSV, with a **100 GB** JSON file in the same bucket waiting to be "just loaded." The reader opened the object whole. The [EMR compare](https://www.factualminds.com/blog/aws-emr-serverless-vs-ec2-vs-eks-cost-comparison/) chooses a deployment mode. The [Glue versus dbt guide](https://www.factualminds.com/blog/aws-glue-vs-dbt-on-aws-data-transformation-guide/) chooses a transformation style. Neither starts from this error. > **Reproduce this:** Copy the [OOM worksheet](https://www.factualminds.com/examples/architecture-blog-2026/large-file-processing/oom-worksheet.md). Write the file size and whether the reader opens it whole before you pick a logo. ## Three shapes **Interactive SQL.** The data is already on S3 and someone needs an answer, not a cluster. Athena reads it. The [Athena cost post](https://www.factualminds.com/blog/athena-query-cost-optimization-partition-compress-cache-iceberg/) owns how not to scan the lake. Do not load the file into a worker to answer one question. **One-shot conversion.** You need the file in a columnar layout once. DuckDB on a single node is the right tool when the object fits that node. It is the wrong tool when you are still on a laptop-sized process and the object is 50 GB or 100 GB. Size the node to the file, or split the file. Do not pretend a laptop success is a design. **A repeating pipeline.** Glue or EMR Serverless, after the input is split into objects the workers read independently. The [lakehouse pattern](https://www.factualminds.com/patterns/lakehouse-on-aws/) owns the platform. The [DuckLake versus Iceberg post](https://www.factualminds.com/blog/ducklake-vs-apache-iceberg-on-aws-2026/) owns the table-format argument. This page does not. We recommend splitting the file before adding workers. The trade-off is a conversion step you did not want. More memory on a job that still opens one object moves the crash to a larger machine. ## Where this fails > **What broke:** Lab, 4 October 2026. Not a customer account. A Glue job with more workers still died on one 50 GB CSV because the read was the whole object. Detection: the worksheet's "opens whole" cell was yes and the fix was a worker count. Recovery: convert to partitioned columnar files, then schedule the pipeline. Lesson: scale is not a substitute for a split. ## What to do this week 1. Find the job that OOMs. Record the object size. Do not record the worker type first. 2. If the need is a query, point Athena at the data and stop. 3. If the need is one conversion, size a single node or split the file. Then use DuckDB or an equivalent one-shot reader. 4. If the need is a pipeline, split first, then Glue or EMR Serverless. 5. A lake that keeps producing this file is a [data analytics](https://www.factualminds.com/services/aws-data-analytics/) conversation, not another worker type. ## What this post doesn't cover It does not compare EMR Serverless with EMR on EC2 or EKS, teach Glue versus dbt, or argue DuckLake versus Iceberg. It does not pick a fan-out for a million small items. That comparison is a different article. ## FAQ ### Glue, EMR Serverless, Athena, or DuckDB when the file does not fit in memory? Start from the shape. Interactive SQL over data already on S3 is Athena. A one-shot conversion of a file that fits one well-sized node is DuckDB on that node. A repeating pipeline is Glue or EMR Serverless after the file is split into pieces the workers can read. Adding workers to a job that opens one 50 GB or 100 GB object does not fix the open. ### When should we not add Glue workers? When the input is one object and the reader loads it whole. More workers multiply the same failed open. Split or convert the file first. Then scale. ### What could go wrong if we use DuckDB because it worked on a laptop sample? The sample fit. The full object does not. DuckDB is the right one-shot tool when the file fits the node you actually run, not the laptop where you tried ten rows. Past that size you need a cluster, and a single-node engine will not give you one. ### Is this the EMR deployment compare? No. The EMR article compares Serverless, EC2, and EKS. Glue versus dbt compares transformation style. This page starts at the out-of-memory error for one large file. ### Where does Athena cost guidance live? On the Athena cost post. This page only says: if the question is interactive SQL, Athena can answer it without loading the file into a worker. It does not restate partition and compression advice. --- *Source: https://www.factualminds.com/blog/large-file-processing-glue-emr-athena-duckdb/*