In short
- Search and answer crawlers decide whether your site can be cited in live answers; training crawlers do not.
- A bot obeys the group that names it, otherwise the wildcard group; the longest matching rule wins and Allow wins a tie.
- robots.txt is one layer: a CDN or firewall can still refuse a crawler it allows.
Three kinds of crawler
Search and answer crawlers (for example OAI-SearchBot and PerplexityBot) fetch pages that engines cite in live answers. Blocking one removes your site from that engine's sources.
User-triggered fetchers (for example ChatGPT-User) fetch a page when a person asks an assistant about it.
Training crawlers (for example GPTBot) collect data for future models. Blocking them is a legitimate choice and does not remove you from live answers.
How a rule is decided
A bot obeys the group that names it, otherwise the wildcard group, otherwise nothing applies. The longest matching rule wins, and Allow wins a tie. A missing robots.txt allows everything.
robots.txt is one layer. A CDN or firewall can still refuse a crawler that robots.txt allows.
Check yours
The free AI crawler check reads a site's robots.txt and shows each crawler's status. It also checks llms.txt and what a crawler sees without running JavaScript.
Run the free AI crawler check on your own site.
Sources
Checked on October 27, 2026.
Keep reading
See your own answers
The free scan asks ChatGPT and Google AI your question several times and shows what each one said, with your brand highlighted. No account needed.
Run the free scan