Skip to main content

Large JSON on AgentCore: Store the Payload in S3, Page the Model (2026)

AI AgentsPalaniappan P3 min read

Quick summary: On 17 November 2026 AWS reported up to 71 percent fewer tokens and 80 percent lower latency on a 23 MB JSON payload by keeping the payload in S3 and showing the model a count plus a handle. This is their benchmark. The failure mode is one tool result that no longer fits the window.

Key Takeaways

  • On 17 November 2026 AWS reported up to 71 percent fewer tokens and 80 percent lower latency on a 23 MB JSON payload by keeping the payload in S3 and showing the model a count plus a handle
  • On 17 November 2026, AWS published a pattern for tool results that do not fit the context window
  • On a 23 MB payload they call Flightscape, they report up to 71 percent fewer tokens and 80 percent lower latency
  • Average latency before the change was above 15 seconds
  • One tool response in that system exceeded 80,000 tokens
Diagram of AgentCore Memory hydrating references, Runtime paging an oversized tool result to S3, and Bedrock receiving a count and a handle
Table of Contents

On 17 November 2026, AWS published a pattern for tool results that do not fit the context window. On a 23 MB payload they call Flightscape, they report up to 71 percent fewer tokens and 80 percent lower latency. Average latency before the change was above 15 seconds. One tool response in that system exceeded 80,000 tokens. Those figures are theirs. We did not run the workload, and this is not an agent case study.

The agent in the pattern reads tool JSON, writes the heavy body to S3, and puts a reference back in AgentCore Memory. The model sees a count and a handle. A sketch of that shape, with no benchmark, is persisted-state.py.

What broke

The same failure shows up without their numbers. If getOrders returns every line item for a year, the next model call pays to reread it. Cap the tool. Page what remains.

The turn

At the start of the turn, Runtime loads references from Memory. Keys such as orders_ref point at S3. They are not the orders.

When a tool returns a body that does not fit, Runtime writes the body to S3 and replaces it with the count and the key. The model can ask for one page. It does not receive the file.

At the end of the turn, Runtime writes the references back to Memory. The next turn hydrates those keys. If you write the JSON into Memory, the next turn loads it again and the S3 copy was wasted.

AWS’s post describes three moves together: S3 for the body, paging for the tool list, and a compact dictionary so repeated keys are not resent. Doing only the dictionary pass leaves the 23 MB body in the prompt.

What this post doesn’t cover

  • Their GitHub companion as a dependency. Read it from the AWS post. Do not treat this page’s sketch as that code.
  • A measured token price. S3 storage and GET requests are cheap next to input tokens, and they are still a bill. Price the object class and the expiry on the AgentCore pricing post.
  • A reason to skip a rule. If the step is a format check, do it in code. The token-budget post has the three questions. This page is only the payload shape after you have decided a model call is required.
  • Memory retention. A 10-year short-term TTL does not make a JSON dump cheaper. Set retention on the memory map.

Sources

Frequently asked questions

Did FactualMinds measure the 71 percent token cut?
No. The 71 percent token reduction and the 80 percent latency reduction are from AWS's 17 November 2026 Machine Learning post, on a 23 MB payload they call Flightscape. We did not run that workload. Use the numbers as their published result, then measure your own tool payloads.
When should we NOT page tool JSON to S3?
Skip it when the tool result already fits the window, when the next step needs every field in one prompt, or when you have no lifecycle for the object. An S3 copy with no expiry is a second store of customer data. A handle the agent cannot read back is worse than the original payload.
When is a smaller model the wrong fix?
When one tool response is tens of thousands of tokens. A cheaper model still has to read the payload. Put the payload in S3 first, then decide whether the remaining judgment step needs a large model. Rules for format checks stay in code. See the token-budget post.
What could go wrong if Memory stores the JSON?
Short-term Memory is the session log, and you pay to keep it. A 23 MB tool result does not belong there. Store a key and a count. The object lives in S3, with a bucket policy and an expiry. Memory that holds the payload will rehydrate it on the next turn and undo the paging.

Reference bounds

How the JSON paging agent stays bounded

Level 2 — reference architecture. Risk medium. Oversight: monitored.

Read a tool result, store an oversized body in S3, and return a count plus a key to the model.

Explore, then act, then confirm, then verify. Explore gathers the context for the JSON paging agent. Act stays reversible. Confirm stops before an irreversible step. Verify is a separate check of the outcome.

Starts when
A tool response is too large for the next model call.
Tools
A write goes through a router, a permission check, a policy check, and a budget check. The model does not commit it. An approval token lives in tool context, not in the user message.
Stops for a person
The agent writes the object and the reference. It does not change the system of record the tool read.
Checked by
The model sees the count and the key. The body stays in S3. A missing key is a failed page, not a guess.
Untrusted data
Tool JSON and any text inside it that tells the agent to ignore the page size.
If it fails
If the JSON paging agent stops, name the reason: completed, budget_exceeded, timed_out, cancelled, guardrail_blocked, approval_required, tool_failure, verification_failed, or partial_completion. Retry a timeout at most twice. Back off on rate limits. After repeated verification failure, escalate. Stop when the budget is exhausted or a permission is denied. No unbounded loop. The agent writes the object and the reference. It does not change the system of record the tool read.

This is the reference architecture for the page, not a published production deployment. The shared contract is the AWS store-agent architecture. Permissions and data boundaries are in securing store agents.

Palaniappan P
Palaniappan P

AWS Cloud Architect & AI Expert

AWS-certified cloud architect and AI expert with deep expertise in cloud migrations, cost optimization, and generative AI on AWS.

AWS ArchitectureCloud MigrationGenAI on AWSCost OptimizationDevOps

Recommended Reading

Explore All Articles »