---
title: Automate Ecommerce Product-Feed Validation on AWS
description: A feed file is not a catalog until it passes the gate. This design quarantines a failed file and waits on a Step Functions task token before release. The large-file cutover is 100 MB, a design choice, not a service limit.
url: https://www.factualminds.com/blog/automate-ecommerce-product-feed-validation-aws/
datePublished: 2026-10-19T00:00:00.000Z
dateModified: 2026-10-19T00:00:00.000Z
author: palaniappan-p
category: Data & Analytics
tags: ecommerce, amazon-s3, aws-step-functions, aws-lambda, aws-glue
---

# Automate Ecommerce Product-Feed Validation on AWS

> A feed file is not a catalog until it passes the gate. This design quarantines a failed file and waits on a Step Functions task token before release. The large-file cutover is 100 MB, a design choice, not a service limit.

On **19 October 2026**, a product file is still the thing that puts a bad price on a product page. The [product-feed glossary](https://www.factualminds.com/glossary/product-feed/) defines what an agent reads. The [catalog article](https://www.factualminds.com/blog/ai-product-catalog-management-ecommerce-2026/) lets a model draft attributes and a merchandiser publish. Neither one stops a dropped CSV. This page is that stop.

> **Reproduce this** — Copy the [release sheet](https://www.factualminds.com/examples/architecture-blog-2026/product-feed-validation/release-sheet.md). Fill the price band before you wire the alarm. A blank band means you have not decided what an unexpected price is.

## Who this is for

Ecommerce operations, and the engineer who receives the file from a PIM, a supplier, or a marketplace export. The file is CSV or JSON in S3. The storefront is a publish target, not the validator. Shopify integration, if you need the platform side, stays on the [Shopify and AWS integration](https://www.factualminds.com/integrations/shopify-aws/).

## What you get, and what you do not

You get a quarantine prefix, a written report, and a release that waits for a person. You do not get a repaired catalog. You do not get a promise that the storefront matches the file after release. Publish is a separate step you already own.

**Opinionated take:** fail the file and wait for the task token. The trade-off is a delayed drop when the checker is wrong. A wrong price on a live product is worse. Tune the band. Do not bypass the gate.

## Architecture

S3 receives the object. EventBridge starts a **Step Functions Standard** workflow, because the approval uses the [callback pattern](https://docs.aws.amazon.com/step-functions/latest/dg/connect-to-resource.html). Express workflows do not support `.waitForTaskToken`, and they stop at five minutes. A person is slower than that.

Lambda validates objects at or under **100 MB**. That cutover is a design choice so the function can stream the file inside a short run. Larger objects are registered as a Glue table and checked with [Glue Data Quality](https://docs.aws.amazon.com/glue/latest/dg/data-quality-getting-started.html). The lakehouse promotion article is the fail-closed swap for a curated table. It is not this release.

```mermaid
flowchart LR
  incoming[S3Incoming] --> eventBus[EventBridge]
  eventBus --> workflow[StepFunctionsStandard]
  workflow --> checker[LambdaOrGlue]
  checker --> quarantine[S3Quarantine]
  checker --> approval[TaskToken]
  approval --> released[S3Released]
```

## Walkthrough

**1. Land the file under `incoming/`.** Turn on EventBridge notifications for the bucket. The rule matches `Object Created` and the prefix. A partner who must use SFTP should land through the Transfer Family workflow into this prefix. Do not build a second server here.

Rule pattern:

```json
{
  "source": ["aws.s3"],
  "detail-type": ["Object Created"],
  "detail": {
    "bucket": { "name": ["feed-incoming-prod"] },
    "object": { "key": [{ "prefix": "incoming/" }] }
  }
}
```

The target is `states:StartExecution` on this state machine only.

**2. Record the attempt before you parse.** The key is bucket, object key, and the S3 ETag from the event. `PutItem` with `attribute_not_exists(attemptId)`. If the item exists and status is `RELEASED`, stop. A retry of the same object must not copy the file twice.

**3. Validate.** For a file at or under 100 MB, a Lambda function streams rows and writes a JSON report object whose name is the ETag. The report lists the first failed checks from the sheet: missing SKU, duplicate SKU, broken variant, currency outside the allowed set, non-numeric or zero price, price delta outside your band, sale start after sale end, image URL that is not https or whose HEAD fails.

For a larger file, create or update a Glue table over the prefix and run a ruleset. DQDL, as documented for column checks:

```text
Rules = [
  IsComplete "sku",
  Uniqueness "sku" = 1.0,
  ColumnValues "currency" in ["USD", "INR", "EUR"]
]
```

`Uniqueness` measures the share of values that appear exactly once. `= 1.0` means every SKU appears once. Price bands and image HEAD requests are not DQDL rules. Keep those in the Lambda path, or accept that the large-file path checks columns only and say so in the report.

**4. Quarantine or ask.** Failures are copied to `quarantine/` and the workflow ends in `FAILED`. Passes call a Lambda with `.waitForTaskToken`. That function emails the report location and the token. The workflow stays open until `SendTaskSuccess` or `SendTaskFailure`, or until `TimeoutSeconds` (this design uses 172800, two days).

Context line: Amazon States Language, Standard workflow, callback integration from the Step Functions developer guide.

```json
{
  "Type": "Task",
  "Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken",
  "Parameters": {
    "FunctionName": "feed-request-release",
    "Payload": {
      "token.$": "$$.Task.Token",
      "reportKey.$": "$.reportKey"
    }
  },
  "TimeoutSeconds": 172800,
  "Next": "CopyToReleased"
}
```

**5. Release.** On success, copy the object to `released/` and set the attempt to `RELEASED`. The storefront job may read only that prefix. It must not read `incoming/`.

## IAM

The start role can call `states:StartExecution` on this state machine. The checker can `s3:GetObject` on `incoming/*`, `s3:PutObject` on `quarantine/*` and `reports/*`, and `dynamodb:PutItem` plus `dynamodb:GetItem` on the attempts table. The approval function can `states:SendTaskSuccess` and `states:SendTaskFailure`. Nothing in this workflow needs `s3:DeleteObject` on `released/`.

Glue, when used, gets `glue:StartDataQualityRulesetEvaluationRun` on the ruleset and read access to the catalogued prefix. It does not get write access to `released/`.

## Failure, retry, idempotency

EventBridge delivery is at-least-once. The attempt item is the idempotency control. Step Functions retries the checker on `Lambda.ServiceException` with a short backoff. It does not retry a validation failure. A validation failure is a result, not a transient error.

A lost approval email is a timeout, not a release. After timeout the attempt is `EXPIRED`. The operator uploads again or sends a new execution. Do not extend the timeout in the console for one file and forget it.

## Cost, as an assumption

[Step Functions Standard pricing](https://aws.amazon.com/step-functions/pricing/) lists **$0.000025** per state transition in US East (N. Virginia) and **4,000** free transitions per month. Assume six transitions per file and 300 files in a month: 1,800 transitions, inside the free tier. Lambda and S3 are separate. [Lambda's published request price](https://docs.aws.amazon.com/whitepapers/latest/big-data-analytics-options/aws-lambda.html) is **$0.20 per million** requests after one million free, plus **$0.00001667** per GB-second after 400,000 GB-seconds. A few hundred short checks sit inside those allowances. Glue Data Quality evaluations are a different bill. Confirm them on the Glue pricing page before you route large files there. This is not a measured saving.

## Testing

Use a fixture file with one duplicate SKU, one zero price, one bad currency, and one image URL that does not resolve. Expect quarantine and no object in `released/`. Then a clean file. Expect the workflow to wait. Call `SendTaskSuccess` with the token and expect one object in `released/`. Send the same S3 event again and expect no second copy.

> **What broke** — Lab, 19 October 2026. Not a customer account. The HEAD check treated HTTP 403 from a hotlink-protected CDN as success because the code only rejected connection errors. Detection: the report row said pass while the storefront image was broken. Recovery: treat any non-2xx HEAD as a failed check, and allow a hostname list for CDNs that block HEAD so those rows use GET with a range. Lesson: a reachable host is not a usable image.

## Production checklist

- Price band and currency list are filled on the sheet, not hard-coded in a second place.
- `released/` is the only prefix the storefront job reads.
- The state machine is Standard, not Express.
- The approval timeout is an explicit number.
- The attempt table point-in-time recovery is on if you need to explain a release later.
- No model is on the path.

## What This Post Doesn't Cover

Attribute extraction and merchandising copy, which belong to the catalog article. Agent-readable completeness, which belongs to the [AI-ready catalog](https://www.factualminds.com/blog/ai-ready-product-catalog-agentic-commerce-2026/). How a partner connects by SFTP. Whether the file is late, which is the [stale-export check](https://www.factualminds.com/blog/detect-stale-ecommerce-data-exports-aws/).

## What to Do This Week

1. Copy the release sheet and write the currency list and the price band.
2. Point one non-production bucket prefix at a Standard workflow that only quarantines.
3. Release the first clean file by sending the task token yourself.

If the file is the easy part and the publish path is the mess, the [data analytics practice](https://www.factualminds.com/services/aws-data-analytics/) is the conversation. Do not start with a model.

## FAQ

### When should we not build this feed gate?
Skip it when the PIM already rejects duplicate SKUs, zero prices, and bad image URLs before anything is exported, and when a person already reviews every file. Also skip it if the business wants a model to rewrite rows. This workflow fails the file. It does not repair it.

### What could go wrong if a passing file is released automatically?
A rule you forgot, such as a sale window in the wrong timezone, reaches the storefront. The task token exists so a person sees the report first. Automatic release is a later change, after the report has been boring for a while.

### Does Glue Data Quality replace the Lambda checks?
Only past the 100 MB design cutover, and only after the object is a catalogued table. Glue Data Quality rules such as IsComplete and Uniqueness check columns. They do not click the storefront, and they do not approve a flash-sale price change.

### What if the same file is delivered twice?
The attempt key is bucket, object key, and ETag. A second event for an ETag already marked released is a no-op. A new ETag is a new attempt, even when the key is unchanged.

---

*Source: https://www.factualminds.com/blog/automate-ecommerce-product-feed-validation-aws/*