Key findings
of the shops shut out an AI answer crawler in robots.txt (ChatGPT search, Claude or Perplexity): 3 of 987. 4.2% shut out a crawler that collects training data.
of the product pages carry a GTIN or MPN, the identifiers that tell an assistant it is the same product in another shop. 41% name only a SKU, 31% no identifier at all.
of the product pages carry ratings in machine-readable form, 70% the brand and 78% the availability.
of the Shopify shops answer with an llms.txt, a file Shopify serves for its stores. On WooCommerce it is 21%.
Who shuts out AI crawlers
A shop decides in its robots.txt which crawlers may read it. AI companies run two kinds: crawlers that fetch pages for answers and search results, and crawlers that collect text to train models. OpenAI, Anthropic and Perplexity keep the two apart, so a shop can stay in the answers and out of the training data. Google’s token Google-Extended covers both for Gemini, training and the grounding of answers in Gemini Apps, and does not affect Google Search.
Answers and search
Gemini, training and answers
Model training
0.3% of the shops shut out at least one of the four answer crawlers, and 4.2% at least one of the two training crawlers. 4.0% block training and let all four answer crawlers in, the split the AI companies describe.
Only a rule that shuts a crawler out of the whole site counts here (Disallow: /). A block in a firewall or CDN, such as Cloudflare’s setting to block AI bots, does not show in robots.txt and comes on top of these numbers.
What assistants find on a product page
On 89% of the shops we found a product page: from Shopify’s public catalog, a product link on the homepage or the sitemaps. 84% of these pages carry Product structured data, the format an assistant reads price, availability and details from without guessing. The product check then read 881 of them field by field.
A SKU identifies a product in its own shop only. A GTIN (the number behind the barcode) or a manufacturer’s part number (MPN) is the same in every shop, which is how an assistant can tell that two offers are one product.
One product page per shop, so these numbers describe a typical page, not a whole catalog. Outside Shopify the page is found through links and sitemaps and can in some shops be a category page. Description and image also count from the page’s meta tags; a description counts by its length, not by what it says.
Sitemaps, llms.txt and structured data
An llms.txt is a short Markdown file at the root of a site that points AI tools at its main pages. 98% of the Shopify shops answer with one, because Shopify serves an “Agent Instructions” file for its stores, about its Shop skill and UCP. On WooCommerce it is 21%, on the other systems 14%.
Large and smaller shops, Shopify and WooCommerce
The less visited shops more often lack identifiers: a GTIN or MPN is on 32% of the product pages among the 100,000 most visited sites and on 23% below them. Between the platforms the brand differs most: 88% of the Shopify product pages name it, 21% of the WooCommerce pages.
| Top 100,000 | 100,001 to 1 million | Shopify | WooCommerce | Other systems | |
|---|---|---|---|---|---|
| Shops checked | 493 | 494 | 554 | 165 | 268 |
| Shut out an answer crawler | 0.2% | 0.4% | 0.2% | 0.6% | 0.4% |
| Shut out Google-Extended | 2.8% | 3.8% | 1.1% | 9.1% | 4.5% |
| Shut out a training crawler | 3.9% | 4.5% | 1.1% | 11% | 6.3% |
| Product page with Product data | 83% | 85% | 91% | 77% | 69% |
| Brand on the product page | 72% | 67% | 88% | 21% | 54% |
| GTIN or MPN on the product page | 32% | 23% | 28% | 3.5% | 43% |
| Sitemap | 97% | 96% | 99% | 98% | 89% |
| llms.txt | 61% | 63% | 98% | 21% | 14% |
| Median score (of 100) | 90 | 94 | 100 | 85 | 80 |
Ranks from the Chrome UX Report: how often Chrome users load a site, not a shop’s revenue. The score is the free store check’s, from robots.txt, product data, sitemap, llms.txt, schema, meta tags and HTTPS.
How we did it
The sites come from Google’s Chrome UX Report (CrUX), which ranks the websites Chrome users open most. We took its list of the top million for October 2026 (CC BY 4.0, from the cache at github.com/zakird/crux-top-lists), counted each site once (shop.com, www.shop.com and de.shop.com are one site) and split it into two groups by rank: the 100,000 most visited sites, and those from 100,001 to 1 million. In each group we went through the sites in a random order and kept the online shops.
A site counted as an online shop when its address points at Shopify’s storefront servers, or when its homepage carries the markup of a shop system: WooCommerce, Magento (Adobe Commerce), Salesforce Commerce Cloud, BigCommerce, Shopware, PrestaShop, VTEX, OpenCart or Ecwid. From the shops found we drew 500 and 500 at random.
Each shop then got the checks of our free AI store readiness checker: the robots.txt rules for seven AI crawlers, the sitemap, llms.txt, structured data and meta tags on the homepage, HTTPS, and one product page, read field by field with the AI product checker. No AI model scored anything.
| Top 100,000 | 100,001 to 1 million | |
|---|---|---|
| Sites looked at | 14,000 | 5,975 |
| Shut out our crawler in robots.txt | 601 | 217 |
| No answer or a server error | 877 | 297 |
| Error page (mostly 403) | 1,774 | 540 |
| Bot check instead of the homepage | 1,456 | 492 |
| No shop system found | 8,650 | 3,686 |
| Online shops found | 642 | 743 |
| Shops drawn | 500 | 500 |
| Shops checked | 493 | 494 |
How our crawler behaved
- It said who it is. Every request carried the user agent AgenticCart-Tools/1.0 with a link to agenticcart.ai/tools.
- It asked first. Before any page of a site it read the site’s robots.txt and left alone whatever that closes to our crawler or to all crawlers.
- It went slowly. At least one second between two requests to the same shop, and slower when a shop answered “too many requests”.
- It kept no content. Only the results of the checks. The list of shops is not published; this page and the download show totals only.
Limits
- robots.txt is one way to block a crawler. Firewalls and CDNs block AI crawlers too, and that can’t be seen from outside, so more shops are closed to AI than these numbers show.
- Shops with their own front end on top of a shop system (headless) show no shop system’s markup and are missing, as are shops that showed us a bot check or an error page.
- Shopify shops were found by their address, the others by their homepage. The platforms’ shares of the sample are no market shares.
- The checks read what is there, not how good it is. A snapshot from November 15, 2026.
Use the data
The totals behind this page are free to use under CC BY 4.0. Please cite “AgenticCart, AI readiness of online shops 2026” with a link to this page.
To see where your own shop stands, run the free store check. It runs the same checks.
Frequently asked questions
How many online shops block AI crawlers?
In our sample of 987 shops from the million most visited websites, 0.3% shut out an AI answer crawler (ChatGPT search, Claude or Perplexity) in robots.txt, and 4.2% a crawler that collects training data. Blocks in a firewall or CDN can’t be seen from outside and come on top.
Does blocking GPTBot keep my shop out of ChatGPT?
No. OpenAI handles its crawlers separately: OAI-SearchBot fetches pages for ChatGPT search, GPTBot collects training data, and a site can allow the first and disallow the second. Anthropic separates Claude-SearchBot and Claude-User from ClaudeBot in the same way.
What product data can an assistant read on a product page?
What the page states in machine-readable form, mostly as schema.org Product data: name, price with currency, availability, image, brand, description, identifiers such as GTIN or MPN, and ratings. In our sample 84% of the product pages carry Product data, but only 27% a GTIN or MPN and 10% their ratings.
Do Shopify stores have an llms.txt?
Most do without doing anything: Shopify serves an llms.txt for its stores, an “Agent Instructions” file about its Shop skill and UCP. 98% of the Shopify shops in our sample answered with one, against 21% of the WooCommerce shops.
How can I check my own shop?
With the free AI store readiness checker at agenticcart.ai/tools/ai-store-readiness. It runs the same checks as this study and shows what to fix.
Can I use these numbers?
Yes. The totals are free to use under CC BY 4.0 with a link to this page. The list of shops is not published.
See how your own shop compares
Enter your shop’s address. The scan shows what ChatGPT, Claude and Gemini can read today and prepares your products for them, in about a minute.
Or start the free 7-day trial right away.