Short definition
AI crawlers are the bots run by AI companies (OpenAI's GPTBot, Anthropic's ClaudeBot, Perplexity's PerplexityBot, Google-Extended) that fetch web content for training and live retrieval.
In depth
AI crawlers fall into two buckets: training crawlers (used to build the model — GPTBot, ClaudeBot, CCBot) and live retrieval crawlers (used at query time — PerplexityBot, OAI-SearchBot, ChatGPT-User). To be cited in AI answers, both should be allowed in robots.txt. Sites that block them lose visibility in the engines that increasingly mediate buyer research. Allowing them is the GEO equivalent of being indexable in classic SEO.
Example
An agency updates robots.txt to explicitly allow 14 AI crawlers and adds /llms.txt. Brand mentions inside ChatGPT and Perplexity rise sharply over the next quarter.