By Lucan Ashvere. Data collected 4 November 2026. Part of our November 2026 AI search visibility research.
Before an AI search engine can quote your pricing page, its crawler has to be allowed to fetch it. So we checked the two files that control (or try to guide) that access, robots.txt and llms.txt, on 145 B2B SaaS websites: the companies we have been studying in categories such as help desks, analytics, billing, developer tools, CMS, scheduling and internal tools. Here is what we found and what we'd put in your own files.
The short version
| What we checked | Result |
|---|---|
| Sites with a readable robots.txt | 137 of 145 |
| Sites serving a Markdown llms.txt at the root | 107 of 145 (74%) |
| robots.txt files that mention at least one AI crawler by name | 27 of 137 (20%) |
| Sites that block OpenAI, Perplexity or Anthropic crawlers from the homepage | 0 |
| Sites that block Common Crawl (CCBot) site-wide | 2 |
| Sites that block ByteDance's Bytespider site-wide | 4 |
| Sites whose rules keep every crawler, Googlebot included, away from everything except the homepage and help pages | 1 |
So most B2B SaaS teams have quietly shipped an llms.txt, while very few have made a deliberate decision about AI crawlers in robots.txt. For 80% of the sites, AI bots simply follow the generic User-agent: * rules.
Which AI crawlers get named
Share of the 137 robots.txt files that address each crawler explicitly (allow or disallow):
| Crawler | Operator and job | Named by |
|---|---|---|
| PerplexityBot | Perplexity, indexing for its answer engine | 25 (18%) |
| GPTBot | OpenAI, collecting data for model training | 24 (18%) |
| Google-Extended | Google, a control token for Gemini training and grounding (not a separate crawler, and not used for Google Search or AI Overviews eligibility) | 23 (17%) |
| ClaudeBot | Anthropic, collecting data for model training | 22 (16%) |
| OAI-SearchBot | OpenAI, indexing for ChatGPT search results | 19 (14%) |
| ChatGPT-User | OpenAI, fetching pages when a user asks ChatGPT to | 19 (14%) |
| Applebot-Extended | Apple, a control token for AI training use | 18 (13%) |
| CCBot | Common Crawl, an open web archive used by many model builders | 15 (11%) |
| Bytespider | ByteDance | 14 (10%) |
| Claude-SearchBot | Anthropic, indexing for search in Claude | 12 (9%) |
Two patterns stand out. First, the sites that name AI crawlers mostly do it to allow them; nobody in the sample blocks the OpenAI, Perplexity or Anthropic bots from the homepage. Second, the search-specific crawlers (OAI-SearchBot, Claude-SearchBot) are named less often than the older training crawlers, even though they are the ones that decide whether you can be found and cited in ChatGPT and Claude answers today.
Why this matters for being recommended
- Training and search are separate decisions. OpenAI, Anthropic and Perplexity each run different user agents for training, for search indexing and for user-triggered fetches. You can, for example, decline training (GPTBot, ClaudeBot, CCBot) and still allow search (OAI-SearchBot, Claude-SearchBot, PerplexityBot). Two sites in our sample do exactly that for Common Crawl.
- Blocking search crawlers takes you out of the answer. If an engine can't fetch your pricing or comparison pages, it will describe you using other people's pages, often the competitor roundups we wrote about in this guide.
- Generic rules can have side effects. One site in the sample disallows everything for all user agents and then re-allows only the homepage and help center. On that domain, Googlebot and every AI crawler can reach only the homepage, assets and help pages. It may be intentional (for example, if marketing pages live elsewhere), but a broad rule like this is worth a second look on any site.
- Google AI Overviews use normal Googlebot access. Google-Extended does not control whether you appear in Search or AI Overviews, so blocking it is not a way to opt out of those, and allowing it doesn't get you in.
What about llms.txt?
llms.txt is a proposed convention (published in 2024) for a Markdown file at your site root that summarises who you are and links to your most useful pages, so language models can read a clean overview. 74% of the sites we checked already serve one. That's a cheap, sensible thing to have. But keep expectations realistic: we're not aware of any major AI search engine that has confirmed it uses llms.txt when choosing what to cite, and our own observations show answers are shaped mostly by ordinary web pages, especially third-party roundups. Treat llms.txt as a tidy summary, not as a ranking lever. The same one-line positioning should appear on your homepage and profiles, where engines definitely read it.
A sensible starting robots.txt for a B2B SaaS site
If you want to be found and cited in AI answers, make it explicit. Adjust the training lines to your own policy:
User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Claude-SearchBot Allow: / # Training crawlers: allow or disallow per your policy User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: * Disallow: /app/ Disallow: /admin/ Sitemap: https://www.example.com/sitemap.xml
Note that a crawler which matches a named group ignores the * group, so repeat any private-path disallows (such as /app/) inside the named groups if those paths are reachable.
A 10-minute check for your own site
- Open yoursite.com/robots.txt. Is any AI search crawler (OAI-SearchBot, PerplexityBot, Claude-SearchBot) disallowed, directly or through a broad
*rule? - Check that /pricing, /blog and your comparison pages are allowed for
*and Googlebot. - Decide your training policy (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended) and write it down explicitly.
- If you publish llms.txt, make sure its one-line description matches your homepage and current positioning, and that its links don't 404.
- Make sure key facts (pricing, plans, integrations) are in crawlable HTML, not only in a JavaScript calculator or a PDF.
Method
On 4 November 2026 we fetched /robots.txt and /llms.txt over HTTPS from each of the 145 company domains. We parsed robots.txt groups and applied longest-match allow/disallow rules for each crawler against /, /blog/ and /pricing. "Named" means the crawler's token appears in a User-agent line. We counted llms.txt only when the file existed and was Markdown (starting with a heading), not an HTML error page. Robots files can change at any time, and robots.txt is a request that well-behaved crawlers follow, not an access control. We don't name individual sites here. If you'd like your own site's result, or think we got something wrong, email us.
Want the full picture?
Crawl access is only the first step; what the engines actually say about you depends on which pages they cite. Our audit runs 30 to 100 real buyer prompts across up to 6 engines, maps the cited sources and checks your crawlability, all in writing with no calls. See packages and pricing (from $690), or email lucan@business-insights.ai for a free 5-prompt check first.