Term

robots.txt

In short

A plain-text file at the root of a site that tells bots which addresses they may visit. Also the most common place where AI crawlers get blocked without anyone noticing.

robots.txt is a text file that lives at yoursite.com/robots.txt. It contains User-agent lines (which bot) and Disallow / Allow lines (which paths). It is a request, not an enforcement: well-behaved bots comply, malicious ones ignore it. All search and AI crawlers comply.

Relationship to AI crawlers

A Disallow: / line under User-agent: * blocks every bot, and therefore GPTBot, ClaudeBot and PerplexityBot too. Some security plugins and CDNs add this line by default, and the site owner doesn't notice. It's also possible to block a specific crawler by name (User-agent: GPTBot / Disallow: /); if that's a conscious choice, fine; if not, it's the quietest way to lose visibility.

Example

User-agent: *
Allow: /
Disallow: /admin/

User-agent: GPTBot
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

Common mistakes

Carrying the Disallow: / line from the development environment into production.
Blocking AI crawlers while keeping Googlebot open and assuming "our SEO is fine".
Declaring an address in the Sitemap: line that doesn't exist.

Checking it

Open the file in a browser; look for the bot names. The AI crawler check at protan.ai does the same job for every AI crawler at once.

Related term: AI crawler