An open, reproducible test suite for AI agents that answer customers on commerce stores. Same fixture store, same probes, published transcripts — so a merchant can tell the difference between an agent that resolves conversations and one that sounds like it does.
Pre-registered. Methodology published; first round not yet run.
Pre-registered on purpose. This page went up before the first round ran. Publishing the tests first is the only way a benchmark run by a company that also builds agents can be worth reading — it means the probes cannot be shaped around a result we would like.
Every vendor is tested against the same seeded catalogue, orders, customers and policies. Each probe runs three times — the worst run is the one that counts, because an agent that is right two times in three is not right.
The single highest-volume intent, and the one most often answered confidently and wrongly.
Policy is easy inside the window. The test is what happens outside it.
Whether the agent will invent a fact when the data layer is silent.
Scored strictly. A single confirmed leak caps the total score regardless of everything else.
Two failure modes, both penalised: escalating what it should handle, and handling what it should escalate.
The pack nobody else tests, drawn from the failure patterns in our 1,140-store cohort.
A wrong delivery estimate annoys a customer. A leaked order history is a breach. The weighting reflects that difference, and a single confirmed privacy leak caps the total regardless of everything else.
Five commitments. They are the whole basis for taking this seriously, so they are stated plainly enough to be held against us.
This page was published before the first round ran. The tests cannot be reshaped around a favourable outcome, and you can check that against the version history.
Our own customer-facing agent is still in development, so it is not in round one and we are not ranked. From the round it ships, it is tested under exactly the same rules — and where it places below other vendors, that is published with the transcript. If we ever quietly skip a round after shipping, stop trusting this.
The probe set and fixture store are open. Anyone can reproduce a round independently and contest the result.
Any vendor may submit a configuration or dispute a score. Re-tests are published alongside the original, not in place of it.
No vendor can pay to be included, excluded, or re-ranked. There is no sponsorship tier and there never will be.
Listed as intent, not as a result — no round has run. Vendors not on this list can ask to be included, and any listed vendor can submit a configuration of their choice.
Submit or request inclusion →Because the methodology was published before the first result, the test cases are open, and any vendor can submit a configuration or dispute a score with the re-test published alongside. We are also not in round one at all — our own customer-facing agent is still in development, so there is nothing of ours to favour. When it ships it enters under the same rules, and where it loses that is published. If those commitments ever lapse, the benchmark deserves to be ignored.
The fixture store is published with the methodology, before any round runs. It is seeded with catalogue, orders and policies drawn from patterns across our 1,140-store cohort rather than from any single client, and every vendor is tested against exactly the same seed.
Because it is the only failure that is not recoverable. A wrong shipping estimate annoys a customer; a leaked order history is a breach. A single confirmed leak caps the total score regardless of performance everywhere else.
Yes. Vendors and merchants running their own agents can both submit. Round one is capped so that every transcript can be reviewed properly rather than sampled.
Yes, to enter and to read. There is no paid tier, no sponsorship, and no charge to be listed.
