---
title: Detect Stale Ecommerce Data Exports before Reports Go Wrong
description: Four sources, one check a day, 120 Scheduler invocations a month. That volume sits inside the 14 million free EventBridge Scheduler invocations. The check notices a missing file. It does not repair the export.
url: https://www.factualminds.com/blog/detect-stale-ecommerce-data-exports-aws/
datePublished: 2026-10-19T00:00:00.000Z
dateModified: 2026-10-19T00:00:00.000Z
author: palaniappan-p
category: Data & Analytics
tags: ecommerce, amazon-eventbridge, amazon-s3, aws-lambda, data-quality
---

# Detect Stale Ecommerce Data Exports before Reports Go Wrong

> Four sources, one check a day, 120 Scheduler invocations a month. That volume sits inside the 14 million free EventBridge Scheduler invocations. The check notices a missing file. It does not repair the export.

On **19 October 2026**, a report is wrong when the file it trusts never arrived, and the dashboard stays green. The [Kinesis pipeline](https://www.factualminds.com/blog/real-time-data-pipeline-kinesis-lambda-dynamodb/) is for events in motion. This page is for the batch export you expect by a clock time.

The Scheduler article tells you to delete a Lambda that only calls one API on a timer. This function is not that wrapper. It lists a prefix, compares metadata to a contract, and publishes a miss. The [legacy batch article](https://www.factualminds.com/blog/legacy-batch-job-modernization-eventbridge-scheduler/) decides where a long job runs. The check itself is short.

> **Reproduce this** — Copy the [freshness contract](https://www.factualminds.com/examples/architecture-blog-2026/stale-exports/freshness-contract.md). One row per source. The schedule expression fires at "late", not at "expected".

## Who this is for

The analytics owner and whoever is blamed when Monday's numbers repeat Sunday's. Sources include the commerce platform, an ERP, a warehouse, and a supplier drop.

## What you get, and what you do not

You get an alarm when an expected object is missing, late, empty, or full of duplicate keys. You do not get a repaired extract. You do not get a curated table. After a good file lands, the promotion gate still decides whether that table moves. A Glue rerun that double-inserts is the bookmark article, not this one.

**Opinionated take:** detect and alert. The trade-off is a human who must rerun the export. A detector that reruns the producer will eventually write Tuesday's file twice.

## Architecture

One Scheduler schedule per contract row. The target is a Lambda. The function reads the row: prefix, expected-by time, minimum bytes, optional distinct-key column. It calls `ListObjectsV2`. A pass emits nothing urgent. A miss publishes a custom metric and notifies SNS.

```mermaid
flowchart LR
  contract[FreshnessContract] --> schedule[EventBridgeScheduler]
  schedule --> checker[ArrivalLambda]
  checker --> bucket[S3Prefix]
  checker --> alarm[MissAlarm]
  bucket --> later[PromotionGate]
```

## Walkthrough

**1. Write the contract before the schedule.** Expected-by is UTC. Minimum bytes is above zero so a touched empty object fails. Owner is a person, not a shared inbox you already ignore.

**2. Create the schedule** with a role that can invoke only this function. Context: AWS CLI `create-schedule`, from the [Scheduler getting-started guide](https://docs.aws.amazon.com/scheduler/latest/UserGuide/getting-started.html). The cron fires when the file is already late.

```bash
aws scheduler create-schedule \
  --name orders-export-late \
  --schedule-expression 'cron(30 6 * * ? *)' \
  --flexible-time-window '{"Mode":"OFF"}' \
  --target '{"RoleArn":"ROLE_ARN","Arn":"FUNCTION_ARN","Input":"{\"source\":\"orders\"}"}'
```

**3. Check the prefix.** Pass when exactly one object in the prefix has `LastModified` inside the window and `Size` at least the minimum. Zero objects is missing. An object older than the window is late. More than one object in the window is unexpected. If the contract names a key column, stream the object and fail when a key repeats. Do not load a multi-gigabyte file into memory. If the object is large, record "duplicate check deferred" and let the promotion gate's `Uniqueness` rule own it. Say that in the alarm so nobody thinks the key check ran.

**4. Do not advance a dataset** from this function. It writes a metric. The promotion job is separate and fail-closed.

## IAM

`s3:ListBucket` on the bucket and `s3:GetObject` on the prefixes in the contract. `cloudwatch:PutMetricData` on the custom namespace you pick. `sns:Publish` on one topic. No `s3:DeleteObject`. No permission to start the source system's job.

## Failure and retries

Scheduler retries the invocation when the target errors, using the retry policy on the target. A validation miss is not an error. Return success and publish the metric, or your retry policy will page twice for one late file. A throttle from S3 is an error. Let Scheduler retry that.

Idempotency: the metric dimension is the source name and the UTC date. A second successful check on the same date overwrites the same point. It does not create a second incident if the alarm is on "missing equals 1".

## Cost, as an assumption

[EventBridge Scheduler pricing](https://aws.amazon.com/eventbridge/pricing/) includes **14 million invocations** a month free, then **$1.00 per million** in the published example. Four sources, once a day, is **120** invocations. That sits inside the free tier. [Lambda](https://docs.aws.amazon.com/whitepapers/latest/big-data-analytics-options/aws-lambda.html) includes **1 million** requests and **400,000** GB-seconds. A list-and-head check stays inside those allowances unless you stream huge objects in the function. Do not do that. Not a measured saving.

## Testing

Point a contract row at an empty prefix and expect a miss. Put a zero-byte object and expect a miss. Put an object with yesterday's timestamp and expect a late result. Put two objects in the window and expect an unexpected-count result. Put one good object and expect no alarm.

> **What broke** — Lab, 19 October 2026. Not a customer account. The schedule ran at the expected time, not after it. The file arrived two minutes later and the alarm fired every morning. Detection: the alarm timestamp matched the producer's success time. Recovery: move the cron to the late mark on the sheet. Lesson: checking at the deadline catches a punctual file as if it were absent.

## Production checklist

- Every source on the sheet has an owner and a late time.
- The function returns success on a business miss.
- Large files do not get a half-finished distinct-key scan presented as a pass.
- The promotion job is still the only writer of the curated table.
- The alarm names the source and the prefix.

## What This Post Doesn't Cover

Streaming ingestion. Glue bookmark recovery. Which compute should run a multi-hour batch. Row-level price and SKU rules on a product feed. Those are the [feed gate](https://www.factualminds.com/blog/automate-ecommerce-product-feed-validation-aws/).

## What to Do This Week

1. Fill one contract row for the export people already distrust.
2. Run the function once by hand against yesterday's prefix.
3. Schedule it at the late time only after that run matches what you saw in the bucket.

When the files are late because the pipeline design is the problem, start with the [data analytics practice](https://www.factualminds.com/services/aws-data-analytics/). A clock will not fix a producer that cannot finish.

## FAQ

### When should we not add an arrival check?
Skip it when the source already pages you, with the same latency you would accept, if the file is missing. Also skip it for a stream. A nightly file that never comes is this page. A Kinesis pipeline is the other article.

### What could go wrong if a late file is treated as today's file?
Yesterday's inventory is labelled with today's date and the report looks current. The contract row has an expected-by time and a minimum size. An object older than the window fails even if the key exists. Do not rename a late object into today's partition to clear the alarm.

### Does this replace Glue Data Quality?
No. This function runs when the file should already be here. The promotion article runs after a batch exists and decides whether the curated table moves. A file can arrive on time and still be wrong. Both checks earn their place.

### Should the Lambda kick the exporter again?
No. A retry from the detector hides a producer that is down and can duplicate a partial file. Alert the owner on the contract. Let that system rerun.

---

*Source: https://www.factualminds.com/blog/detect-stale-ecommerce-data-exports-aws/*