AI Crawler
An AI crawler is a bot that fetches web content for AI systems — either to build training corpora (GPTBot, ClaudeBot) or to retrieve live pages for grounded answers (PerplexityBot, OAI-SearchBot). Whether you allow or block each crawler in robots.txt determines whether your content can shape AI training data and real-time answers.
AI crawlers split into two functional classes that deserve separate decisions. Training crawlers collect content for future model versions — blocking them keeps you out of tomorrow's parametric knowledge. Retrieval crawlers fetch pages at answer time for engines that cite sources — blocking them removes your pages from today's grounded answers, often handing the citation to a third party that describes you less accurately.
Many sites blocked AI crawlers reflexively during the early scraping debates and never revisited the decision. For a B2B SaaS brand whose growth depends on being found and described correctly, blocking retrieval crawlers is usually self-harm: the engines still answer questions about you — from other people's pages.
Audit your robots.txt against the current list of AI user agents, decide per class (training vs retrieval) rather than blanket-blocking, and verify in server logs that the crawlers you allow actually reach your key pages.
Related terms
See what AI currently says about your brand — free snapshot.