AI Crawler
The crawler programs that language model companies use to read web pages – GPTBot, ClaudeBot, PerplexityBot, Google-Extended and others. If one of them can't get into your site, that engine can't learn about you.
An AI crawler is an automated crawler that a language model company runs to read the web. It works on the same logic as search engine bots (Googlebot, Bingbot): it requests pages, reads them, stores them. The difference is that what it reads goes not into a search index but into a model's training data or its live answer sources.
The main crawlers
- GPTBot — OpenAI, for training data. OAI-SearchBot — for ChatGPT's search feature.
- ClaudeBot — Anthropic.
- PerplexityBot — Perplexity; for showing pages as sources in answers.
- Google-Extended — Google; for training Gemini. A separate permission from Googlebot.
- Bytespider, CCBot, Applebot-Extended and others.
Why it matters
If a crawler can't get into your site, the corresponding engine either never learns about you or only knows as much as others say about you. Many sites block every AI crawler without realising it, through the default setting of a security plugin or CDN.
How to allow access
In robots.txt, a User-agent line for each crawler with Allow: / beneath it; or remove the Disallow lines that block them. On the CDN side (Cloudflare, for example), make sure the "AI bots" rule is switched off. The AI crawler check at protan.ai reads your robots.txt and shows which crawlers can get in, all at once.
Related term: robots.txt