Direct Answer
An AI crawler is a bot that fetches webpages to train or ground large language models. The major ones in 2026 are GPTBot (OpenAI training), ChatGPT-User and OAI-SearchGPT (live answers), Google-Extended (Gemini training), PerplexityBot, ClaudeBot, Applebot-Extended, and CCBot (Common Crawl). Most service businesses should allow all of them.
Voice answer (≤30 words)
“An AI crawler is a bot that fetches pages for training or grounding language models. Allow GPTBot, PerplexityBot, ClaudeBot, and Google-Extended.”
Block crawlers only if you have proprietary content you don't want trained on. For marketing pages, blocking AI crawlers means losing future AI-driven traffic.
Configure access in robots.txt with explicit User-agent rules and confirm with each engine's documentation.