Agentic Commerce Breakage Index Methodology

The Breakage Index measures whether AI shopping answers match real store evidence. It is a fidelity benchmark, not an AI visibility ranking.

A store counts as broken for a metric only when at least one confirmed finding survives reproducibility and calibration gates.

Public release: Summer 2026. Frozen prompt content hash: 3af595138de35b2a5655e8a2234ebdeddc915a46be62b517e73048e89e382d40.

The public prompt table preserves exact prompt text, model IDs, parser schemas, and audit fields; internal protocol IDs remain available through the run provenance.

System prompt: You are evaluating one public ecommerce store for StoreSteady. Answer only from the store evidence and current AI shopping answer context supplied in the prompt. If evidence is missing, say it is missing. Do not infer private, draft, or admin-only state from public absence.

Scan window: 2026-05-23T16:50:03.820Z to 2026-05-23T23:24:45.338Z.

Response audit fields: prompt_version, system_prompt_version, model_id, provider, agent_surface, timestamp, raw_response_hash, parsed_response, parser_schema_version.

The External Top-500 floor is a fast-launch compromise: large enough for a credible inaugural benchmark, with uncertainty disclosed for every headline and agent cut.

Per-agent cuts can publish only with their own denominator and uncertainty disclosure; they do not replace the all-store headline denominator.

For the Summer 2026 report, per-agent cuts are filterable data only. They are not used for launch copy, vendor rankings, or agent-superlative claims.

Summer 2026 public artifacts report all-store cuts only. Per-vertical cuts were withheld because the vertical validation review did not pass the publication threshold.

The External Top-500 corpus uses Tranco list VNGQN from 2026-05-21, filtered to retail and Shopify-detected domains. Included stores: 500. The internal corpus is disclosed separately and is never blended into the External Top-500 headline number. Canonical path: /resources/research/agentic-commerce-breakage-index.

The internal StoreSteady scan corpus is self-selected and reported with a self-selection disclosure when available.

The headline unit is the store, not the finding and not the response.

Each metric publishes its own evaluable_store_n. Stores with no observable truth for a metric are excluded from that metric denominator and counted in the exclusion funnel.

The at-least-one-breakage headline uses stores evaluable for the full metric bundle, or explicitly discloses the minimum metric-evaluable threshold used.

Crawler blocks, robots restrictions, password or challenge gates, and other non-circumvented access controls are counted separately as access_friction_observed. They are not counted as product, policy, offer, checkout, hidden-product, or competitor breakage unless the relevant detector also has enough observable evidence.

StoreSteady does not bypass anti-bot systems, login walls, payment gates, robots restrictions, or challenge pages during this benchmark.

A store can remain in the research frame while a specific detector job is not evaluable. The scan-limitations table separates access_friction_observed from not_evaluable_for_detector so readiness signals do not inflate breakage rates.

Access friction is merchant-actionable product value for Pro Scan because agent-like systems may fail to read, compare, or recommend a store even when no detector-specific mismatch can be evaluated.

Each metric uses the same public tolerance bands throughout the Summer 2026 report.

The calibration set samples flagged candidates and non-finding candidates. Public calibration rows expose labels and rubric versions only, not raw store names, raw URLs, or raw AI responses.

Two independent LLM judges score each candidate. Disagreements require human tiebreak. Krippendorff alpha is published per metric.

Headline metrics require alpha at least 0.80 and calibration accuracy at least 95%. Alpha from 0.667 to 0.80 is tentative. Alpha below 0.667 is dropped.

Public evidence examples are derived from normalized calibration evidence packages. A fail-closed safety review blocks any example containing store names, merchant domains, raw URLs, source quotes, raw AI responses, exact promo codes, or merchant-identifying details.

Judge 1 model: gpt-4.1. Judge 2 model: claude-sonnet-4-6.

Rubric versions: product_truth_mismatch: product_attribute_v1, product_attribute_v2, policy_misstatement: policy_free_text_v1, policy_free_text_v2, offer_ambiguity: offer_ambiguity_v1, offer_ambiguity_v2, non_public_product_exposure: non_public_product_exposure_v1, non_public_product_exposure_v2, checkout_exception_risk: checkout_exception_risk_v1, checkout_exception_risk_v2, competitor_in_branded_prompt: branded_competitor_v1, branded_competitor_v2.

Two independent LLM judges label calibration rows; disagreements are resolved using the human-authored labeling policy and persisted as human tiebreak labels.

The Breakage Index measures answer fidelity and access friction. It does not measure conversion impact, revenue loss, refund rate, chargeback rate, or support-ticket volume.

This report reflects stores that were evaluable inside the report window. Each metric publishes its own evaluable_store_n, and stores with missing truth are counted in the exclusion funnel instead of being treated as breakage.

Stores that block crawler or agent-like access are retained in the research frame when they are otherwise eligible retail Shopify storefronts. Blocked access is reported separately as access friction, not as a detector mismatch.

The internal corpus consists of stores that voluntarily ran a StoreSteady scan during the reporting period. It over-represents operators who already suspected AI-search issues and is never blended into the External Top-500 headline number.

The External Top-500 is drawn from a frozen Tranco snapshot, filtered to retail and Shopify-detected domains. Long-tail, non-English, and non-Shopify stores are under-represented by design and disclosed as out of scope for this inaugural index.

Methodology gaps, reproducible errors, or source corrections can be sent to research@storesteady.com. Corrections are published with the version history below.