AI Crawlers and llms.txt: How 145 B2B SaaS Sites Handle Them (November 2026)

By Lucan Ashvere. Data collected 4 November 2026. Part of our November 2026 AI search visibility research.

Before an AI search engine can quote your pricing page, its crawler has to be allowed to fetch it. So we checked the two files that control (or try to guide) that access, robots.txt and llms.txt, on 145 B2B SaaS websites: the companies we have been studying in categories such as help desks, analytics, billing, developer tools, CMS, scheduling and internal tools. Here is what we found and what we'd put in your own files.

The short version

What we checkedResult
Sites with a readable robots.txt137 of 145
Sites serving a Markdown llms.txt at the root107 of 145 (74%)
robots.txt files that mention at least one AI crawler by name27 of 137 (20%)
Sites that block OpenAI, Perplexity or Anthropic crawlers from the homepage0
Sites that block Common Crawl (CCBot) site-wide2
Sites that block ByteDance's Bytespider site-wide4
Sites whose rules keep every crawler, Googlebot included, away from everything except the homepage and help pages1

So most B2B SaaS teams have quietly shipped an llms.txt, while very few have made a deliberate decision about AI crawlers in robots.txt. For 80% of the sites, AI bots simply follow the generic User-agent: * rules.

Which AI crawlers get named

Share of the 137 robots.txt files that address each crawler explicitly (allow or disallow):

CrawlerOperator and jobNamed by
PerplexityBotPerplexity, indexing for its answer engine25 (18%)
GPTBotOpenAI, collecting data for model training24 (18%)
Google-ExtendedGoogle, a control token for Gemini training and grounding (not a separate crawler, and not used for Google Search or AI Overviews eligibility)23 (17%)
ClaudeBotAnthropic, collecting data for model training22 (16%)
OAI-SearchBotOpenAI, indexing for ChatGPT search results19 (14%)
ChatGPT-UserOpenAI, fetching pages when a user asks ChatGPT to19 (14%)
Applebot-ExtendedApple, a control token for AI training use18 (13%)
CCBotCommon Crawl, an open web archive used by many model builders15 (11%)
BytespiderByteDance14 (10%)
Claude-SearchBotAnthropic, indexing for search in Claude12 (9%)

Two patterns stand out. First, the sites that name AI crawlers mostly do it to allow them; nobody in the sample blocks the OpenAI, Perplexity or Anthropic bots from the homepage. Second, the search-specific crawlers (OAI-SearchBot, Claude-SearchBot) are named less often than the older training crawlers, even though they are the ones that decide whether you can be found and cited in ChatGPT and Claude answers today.

Why this matters for being recommended

What about llms.txt?

llms.txt is a proposed convention (published in 2024) for a Markdown file at your site root that summarises who you are and links to your most useful pages, so language models can read a clean overview. 74% of the sites we checked already serve one. That's a cheap, sensible thing to have. But keep expectations realistic: we're not aware of any major AI search engine that has confirmed it uses llms.txt when choosing what to cite, and our own observations show answers are shaped mostly by ordinary web pages, especially third-party roundups. Treat llms.txt as a tidy summary, not as a ranking lever. The same one-line positioning should appear on your homepage and profiles, where engines definitely read it.

A sensible starting robots.txt for a B2B SaaS site

If you want to be found and cited in AI answers, make it explicit. Adjust the training lines to your own policy:

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

# Training crawlers: allow or disallow per your policy
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: *
Disallow: /app/
Disallow: /admin/

Sitemap: https://www.example.com/sitemap.xml

Note that a crawler which matches a named group ignores the * group, so repeat any private-path disallows (such as /app/) inside the named groups if those paths are reachable.

A 10-minute check for your own site

  1. Open yoursite.com/robots.txt. Is any AI search crawler (OAI-SearchBot, PerplexityBot, Claude-SearchBot) disallowed, directly or through a broad * rule?
  2. Check that /pricing, /blog and your comparison pages are allowed for * and Googlebot.
  3. Decide your training policy (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended) and write it down explicitly.
  4. If you publish llms.txt, make sure its one-line description matches your homepage and current positioning, and that its links don't 404.
  5. Make sure key facts (pricing, plans, integrations) are in crawlable HTML, not only in a JavaScript calculator or a PDF.

Method

On 4 November 2026 we fetched /robots.txt and /llms.txt over HTTPS from each of the 145 company domains. We parsed robots.txt groups and applied longest-match allow/disallow rules for each crawler against /, /blog/ and /pricing. "Named" means the crawler's token appears in a User-agent line. We counted llms.txt only when the file existed and was Markdown (starting with a heading), not an HTML error page. Robots files can change at any time, and robots.txt is a request that well-behaved crawlers follow, not an access control. We don't name individual sites here. If you'd like your own site's result, or think we got something wrong, email us.

Want the full picture?

Crawl access is only the first step; what the engines actually say about you depends on which pages they cite. Our audit runs 30 to 100 real buyer prompts across up to 6 engines, maps the cited sources and checks your crawlability, all in writing with no calls. See packages and pricing (from $690), or email lucan@business-insights.ai for a free 5-prompt check first.