AI Crawler Access: The robots.txt Decisions That Shape Your AI Visibility
· 6 min read · By Perciva Team
Your robots.txt now controls three separate relationships with AI: whether your content trains future models, whether it appears in AI search results, and whether an AI assistant can fetch a page when a user asks about you. Each is governed by different user-agents, and blocking the wrong one silently removes you from answers your buyers are reading.
For most B2B SaaS companies the strategic answer is simple: allow AI crawlers on your public marketing and documentation pages, because being absent from AI answers is a demand-generation problem, not a content-protection win. But "allow" should be a decision, not a default you never examined. Here is the current crawler roster, what each user-agent actually does, and correct configurations for both open and selective postures.
Three Crawl Purposes, Three Decisions
- Training crawlers collect content to train future models. Blocking them is a bet that protecting content outweighs being known by the models buyers use.
- Search-index crawlers build the retrieval indexes behind AI search products. Blocking these directly removes you from cited, real-time answers.
- User-triggered fetchers retrieve a page live because a user asked. Blocking these means when a buyer says "check Acme's pricing page," the assistant cannot.
These are independent controls. The most common misconfiguration is blocking all three when the intent was only the first.
The Crawler Roster
OpenAI
- GPTBot — training. Blocking it signals your content should not train OpenAI models. It does not affect ChatGPT search visibility.
- OAI-SearchBot — indexing for ChatGPT search. Block this and you disappear from ChatGPT's cited web results.
- ChatGPT-User — user-triggered fetches during conversations. OpenAI notes user-initiated fetching may not behave like automated crawling.
Anthropic
- ClaudeBot — training.
- Claude-SearchBot — search indexing.
- Claude-User — user-triggered fetches.
Anthropic documents all three as respecting robots.txt, and each requires its own directive — blocking ClaudeBot alone does not block the other two. More detail in our AI crawler glossary entry.
Perplexity
- PerplexityBot — builds Perplexity's search index. Blocking it removes you from one of the most citation-forward engines B2B buyers use.
- Perplexity-User — fetches pages when a user asks. Perplexity states that because these are user-initiated requests, robots.txt directives generally do not apply to them.
Google
- Googlebot — classic search crawling, which also feeds AI Overviews and AI Mode.
- Google-Extended — not a separate crawler but a robots.txt control token. Disallowing it opts your content out of Gemini training and Search-grounded Gemini answers. Two facts teams routinely get wrong: blocking Google-Extended does not remove you from AI Overviews (those ride on normal Google indexing), and Google has stated it does not affect Search rankings.
Others worth knowing
CCBot (Common Crawl) feeds public datasets used in many models' training. Applebot-Extended, meta-externalagent, Amazonbot, and ByteDance's Bytespider follow the same pattern of vendor-specific tokens — audit them when your policy needs to be exhaustive rather than pragmatic.
Configuration 1: The Open Posture (Recommended for SaaS)
User-agent: *
Disallow: /app/
Disallow: /api/
Disallow: /account/
Sitemap: https://acme.com/sitemap.xml
Everything public stays crawlable for everyone; only the logged-in product, API, and account surfaces are excluded. Two housekeeping notes that save real pain: keep staging and preview environments behind authentication or a blanket disallow, because a crawled staging site with placeholder pricing can leak into answers; and remember that robots.txt is per-host, so your docs subdomain needs its own correct file. If you maintain an llms.txt file, verify every URL it lists is allowed here — the two files must agree.
Configuration 2: Selective — Search Yes, Training No
Some companies (usually content-heavy ones) want AI search citations without contributing training data:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: *
Disallow: /app/
Understand the cost before copying this: models trained without your content know less about you, and what they "know" comes entirely from third parties. For a niche SaaS brand, that often means being described by your competitors' comparison pages. Being retrievable at answer time softens this, but training-time familiarity still influences which brands models reach for — being absent from it is a real trade, and for most vendors a bad one.
robots.txt Is Not the Whole Story
- Your CDN may be blocking AI crawlers for you. Bot-management defaults at several major CDNs challenge or block AI user-agents regardless of robots.txt. If your visibility is mysteriously poor, check the WAF before rewriting content.
- robots.txt is a request, not a wall. Reputable crawlers comply; disreputable ones do not. Enforcement requires network-level controls.
- Verify with logs. Your server logs show which AI user-agents actually fetch which pages and how often. Docs and pricing pages drawing steady fetches from search-index bots is what healthy looks like.
Verify in the Logs, Not the Config
Your robots.txt states policy; your access logs state reality. A quick check on most setups:
grep -iE "gptbot|oai-searchbot|chatgpt-user|claudebot|claude-searchbot|perplexitybot" access.log | awk '{print $1, $7, $12}' | sort | uniq -c | sort -rn | head -50
Healthy looks like steady fetches from search-index bots across your docs, pricing, and comparison pages. Two anomalies worth chasing: a bot you allowed that never appears (check CDN bot management — it may be challenged before reaching your origin), and heavy fetching of pages you disallowed (either a misconfigured rule or a spoofed user-agent; major vendors publish IP ranges so you can verify authenticity before accusing anyone of noncompliance).
Frequently Asked Questions
If we block GPTBot, do we disappear from ChatGPT?
No — and this is the most consequential nuance on the page. Blocking GPTBot only opts you out of future training. ChatGPT can still cite you via search (OAI-SearchBot) and still "knows" whatever earlier training absorbed. Conversely, allowing GPTBot does not get you into ChatGPT search — that is OAI-SearchBot's job. Match the user-agent to the outcome you actually care about.
Should we block Bytespider and other aggressive crawlers?
ByteDance's Bytespider has drawn persistent complaints about crawl aggression, and crawlers with poor compliance reputations make robots.txt directives unreliable as enforcement. If you decide to block a crawler you do not trust, do it at the WAF or network level and treat the robots.txt line as documentation of intent.
Does any of this affect our Google rankings?
Blocking AI-specific tokens like Google-Extended does not affect classic Search rankings — Google has said so explicitly. Blocking Googlebot itself, of course, is a different and much larger decision.
The Decision Checklist
- Decide the three postures explicitly: training, search indexing, live fetch.
- Never block search-index bots (OAI-SearchBot, PerplexityBot, Claude-SearchBot) on pages you want cited — see how to get cited by Perplexity.
- Keep pricing, security, docs, and comparison pages crawlable — they answer the buyer questions with money attached.
- Audit CDN and WAF rules against your robots.txt intent.
- Re-run the audit quarterly; vendors add user-agents without ceremony.
Access Is the Prerequisite, Not the Prize
Correct crawler access makes you quotable; it does not make you quoted. Once the gates are open, the question becomes which sources engines actually cite when buyers ask about your category — and whether any of them are yours. That is citation territory: measure it, then close the gaps. Perciva tracks which pages AI engines cite on real buyer questions, so you can see whether opening the gates changed the answers.