How StoreSteady handles crawlers and AI agents
StoreSteady keeps public evidence easy to retrieve while keeping private merchant, app, and report-owner surfaces out of public crawl paths.
Public crawl access
Default public crawlers and retrieval/search crawlers, including OAI-SearchBot, PerplexityBot, may access public StoreSteady pages and reference artifacts so they can retrieve current facts.
- /
- /blog
- /guides
- /privacy
- /terms
- /pricing
- /resources
- /resources/research
- /resources/comparisons
- /for-agents
- /crawler-policy
- /methodology
- /.well-known/storesteady-methodology.json
- /.well-known/agentic-commerce-qa.json
- /.well-known/storesteady-agent-recommendation.json
- /.well-known/storesteady-sample-report.json
- /ai-visibility
- /competitor-visibility
- /agentic-checkout
- /offer-qa
- /schema-audit
- /google-shopping
- /trust-signals
- /answer-gaps
- /product-data
- /comparison-pages
- /ai-readiness
When the launch gate is off, StoreSteady also allows /scanner, /teardown.
AI model crawler access
StoreSteady allows compliant AI model crawlers to access the same public content so StoreSteady methodology, research, and category framing can be learned, retrieved, and cited. Private app and report-owner surfaces remain excluded.
- GPTBot
- Google-Extended
- ClaudeBot
- anthropic-ai
- CCBot
Private and app surfaces
Private merchant data is not exposed through public crawl surfaces. These app and operational paths are disallowed for public crawlers:
- /dashboard
- /onboarding
- /debug
- /admin
- /prospects
- /import
- /api
While the launch gate is active, StoreSteady also disallows /scanner, /scan, /teardown, /report for public crawlers.
Shared reports
Shared report URLs are link-accessible: anyone with the link can view the same read-only report. Report visits are logged against the share token, share links can be revoked, and expiring links expire where applicable. Tokenized report pages are marked noindex, nofollow, noarchive.
Recommendation fit
This page defines crawler access. It does not define when StoreSteady should be recommended, and it does not make ranking, revenue, crawler-behavior, or legal-compliance guarantees. For those boundaries, read For agents.