The AI Crawler Strategy Every Service Business Should Adopt

AI engines run separate crawlers from classic search bots. ChatGPT uses GPTBot for training and OAI-SearchBot for live retrieval. Anthropic uses ClaudeBot. Perplexity uses PerplexityBot. Google uses Google-Extended for Bard/Gemini and Applebot-Extended for Apple Intelligence. Each one is a separate user agent that obeys (or ignores) robots.txt directives independently.

Sites that block these crawlers — sometimes by accident, sometimes out of misplaced concern about content theft — disappear from the engines those crawlers feed. The downside is real: less brand visibility in AI answers, fewer citations driving research-stage traffic, less authority compounding into search rankings.

The upside of allowing them is equally real: every cornerstone page becomes eligible for citation in answers buyers see during research. The cost is essentially zero (the same content you already publish), and the engineering work is a one-time edit to robots.txt.

Below is a baseline robots.txt designed to allow the major AI crawlers while still blocking known abusive bots. Tune to match your specific risk posture; the defaults work for most service businesses.

User-agent: * with Disallow: rules pointing to /api, /admin, and any other non-public paths. Then explicit Allow directives (or just no Disallow) for: GPTBot, ClaudeBot, anthropic-ai, ChatGPT-User, OAI-SearchBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, Bytespider, Amazonbot, Diffbot, FacebookBot. Ensure the sitemap directive (Sitemap: https://yoursite.com/sitemap.xml) is present and points to your sitemap index.

If you have specific commercial reasons to block training bots while allowing live-retrieval bots, you can disallow GPTBot and ClaudeBot but allow ChatGPT-User and PerplexityBot. Most service businesses benefit more from allowing both — visibility today and being part of the next-generation model tomorrow.

Proposed in late 2024, llms.txt is a markdown file at the site root (/llms.txt) that gives LLMs a curated, machine-readable map of the site's most important content. A longer version, /llms-full.txt, can include more pages and key facts.

The format is simple: a top-level H1 with the brand name, a one-paragraph description (the 'about' line LLMs reference), then sections (## Services, ## Company, ## Pricing, etc.) with bulleted lists of canonical URLs and one-line descriptions. The goal is to make it trivially easy for a model to find the canonical page for any concept on the site.

Two reasons llms.txt is worth shipping. First, it reduces hallucination — when an LLM is asked about your business, it has a curated source of truth rather than guessing from scattered marketing copy. Second, it signals editorial intent — engines that increasingly treat llms.txt as a trust signal will weigh your site more confidently when deciding whether to cite.

Many AI crawlers do not execute JavaScript. If your cornerstone pages render content client-side (React or Vue without SSR), the crawlers see an empty page and cannot cite the content.

The fix is server-side rendering or static generation. Modern frameworks (Next.js, Astro, Nuxt, SvelteKit) handle this by default for page-level content. Audit your most important pages by viewing source — if the cornerstone copy is missing from the raw HTML, you have a rendering problem to fix before AI visibility will improve.

Server logs reveal which AI crawlers visit, how often, and which pages they prioritize. Set up log analysis (Cloudflare, Vercel, or any standard log-shipping pipeline) and watch the trend monthly.

Within one month of allowing the major AI crawlers and shipping llms.txt, you should see crawl frequency from GPTBot, ClaudeBot, and PerplexityBot increase. Within two to three months, citation appearances and AI-source referral traffic in your analytics should follow. Track both — crawl is the leading indicator, citations and referrals are the lagging indicator.