Skip to main content

AWS Cloud Architect & AI Expert

Palaniappan P

AWS-certified cloud architect and AI expert with deep expertise in cloud migrations, cost optimization, and generative AI on AWS.

Published articles
354
AWS certifications
3
Palaniappan P

AWS Cloud Architect & AI Expert

View LinkedIn profile

Expertise

AWS ArchitectureCloud MigrationGenAI on AWSCost OptimizationDevOps

Certifications

  • AWS Solutions Architect Professional
  • AWS Security Specialty
  • AWS DevOps Engineer
A 50 GB CSV opened whole runs out of memory; Athena for questions, DuckDB for one conversion, Glue or EMR Serverless for pipelines, after splitting the input
Data & Analytics

When the Data Job Runs Out of Memory

A 50 GB or 100 GB CSV loaded whole dies in a small worker. Split the job into interactive SQL, a one-shot conversion, or a repeating pipeline. Adding workers does not split a file nobody partitioned.

2 min read

More articles

Work With Us

Our articles share what we know. Our consulting engagements apply that knowledge to your specific AWS environment.