AI Agent Vendor Evaluation Checklist for eCommerce (2026)

On 25 September 2026, most “agent” pitches to eCommerce teams are still a chat window on an admin credential. This checklist is how a buyer tells them apart before a pilot. It does not name a winning vendor. FactualMinds sells an implementation engagement; treat that as a conflict and score us on the same lines.

Who this is for. Someone comparing implementers or platforms for one store workflow. If you have not chosen the workflow, use how to evaluate AI agent opportunities first. That post scores the work. This one scores the supplier.

Our take: if they cannot show a denied write, they are selling a chatbot. Walk away even if the demo was fluent.

How to use it

Score each line pass / gap / fail. A single fail on security, ownership, or evaluation blocks a production pilot. Gaps can be a fixed-scope remedy. Do not average the lines into a vanity total.

Copy the same headings into the RFP template.

Your own readiness still gates the project. Below 16 out of 30, do not let any vendor attach a write. The rubric is the readiness assessment.

1. Architecture

• One workflow, named. Not “the commerce agent.”
• A diagram that shows the model, the tool boundary, and the system of record.
• Hosting you can point at (your AWS account, or a tenant they will describe). AgentCore Harness has been GA since 17 June 2026. Agents Classic is the wrong net-new path after 30 July 2026. A vendor still demoing Classic for a new build is behind.
• No claim of a native Shopify–AgentCore connector. There isn’t one. Tools are OpenAPI, MCP, or Lambda you or they own.

2. Security

• Tool allow-list and a deny-list for refunds, cancels, price writes, and inventory adjusts.
• Authorization outside the prompt. Prompt text is not a boundary. See securing agents on the store.
• Separate credentials from any human admin.
• A trace of one blocked write, produced in the demo, not described.

3. Integration

• The API version they will pin (Shopify Admin, Adobe Commerce /rest/V1/, BigCommerce Management API, or your custom OMS).
• Webhooks or polls, and what happens when the downstream API times out: stop, do not invent an order status.
• Idempotency if a write exists at all.

4. Data

• Which fields the agent may see. Email and full address are a decision, not a default.
• Join keys written down (shopify_order_id to OMS id, or the equivalent).
• Stale inventory and stale price called out as a known failure, with an asOf timestamp or an escalate rule.

5. Evaluation

• A golden set you can add to: at least ten lookups and three must-escalate cases. Method: ten tickets.
• A pass bar written in advance. “Looks good” is not a bar.
• Regression on every prompt or tool change. A vendor who retunes in production without a re-run fails this line.

6. Observability

• Per turn: tools called, policy decision, latency, and cost.
• Export. A portal you cannot leave is a gap, not a pass.
• An alarm when a forbidden tool is invoked.

7. Governance

• Named human for anything that moves money, inventory quantity, or a customer-specific price.
• Autonomy chosen per action, not “level 5” for the bot. The spectrum is how much autonomy.
• Audit retention stated in days, and who can read it.

8. Cost

• Split worksheet: platform, model, build, eval, your reviewers. Method: what an agent costs.
• They can explain the published floor — about $791 per month at 50,000 sessions for one AgentCore support silhouette in the July 2026 benchmark — and why their number differs.
• A cap that stops the loop. Browser or code-interpreter on every turn is a cost defect, not a feature.

9. Scalability

• What changes at 10× sessions: model quota, Gateway limits, human queue. If the answer is “the model scales,” they skipped the queue.
• One store first. Multi-brand is a second design.

10. Support

• Severity definitions and who is paged after go-live.
• They will say what they will not operate. A vendor who implies 24×7 autonomous refunds fails the honesty test.

11. Ownership

• You keep tool schemas, policies, eval data, and infrastructure-as-code.
• A prompt export is not ownership.
• Your orders are not training data. Default no. Get it in the contract.
• Exit in the SOW, not in a renewal conversation.

What a fail looks like

What broke — A pilot installed a “read-only” app whose token still included order-write scope because that was the preset. The model did not refund anyone in week one. The capability was already there. Detection: the granted scopes listed write. Fix: reinstall with read scopes only; delete write tools from the schema. Lesson: scope is the boundary. The prompt is a wish.

If you only do one thing

Ask for the allow-list and a denied-write trace. If the meeting moves on to a roadmap slide, you already have your answer.

What to do this week

1. Name one workflow and the system of record.
2. Score readiness. Under 16, stop.
3. Send this checklist with the RFP.
4. Compare build, buy, and partner-build only after a vendor survives sections 2, 5, and 11. The frame is build vs buy.
5. If you want that review done against your stack, talk to FactualMinds about the opportunity. We will say no when the score says no.

What this post doesn’t cover

• Which workflow to pick. That is the opportunity post and which agent first.
• A ranked vendor directory. We are not publishing one.
• Legal review of a contract. Have counsel read ownership and training clauses.
• Proof that any supplier, including us, has a published commerce-agent case study. We do not.