Free tool · No signup
AI Crawler Access Checker
Paste a URL to see which of 17 AI crawlers your robots.txt allows or blocks, and whether you're blocking AI search by mistake.
Training crawlers vs. search crawlers
AI companies run different bots for different jobs, and robots.txt lets you treat them differently:
- Training crawlers (such as GPTBot, ClaudeBot, CCBot, Google-Extended) collect content that may be used to train models. Blocking them does not remove you from AI search answers.
- Search crawlers (such as OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot, Bingbot) index pages so they can be found and cited in answers. Blocking them can make you invisible in AI search.
- User-requested fetchers (such as ChatGPT-User, Claude-User, Perplexity-User) load a page when a person asks an assistant about it.
Crawlers this tool checks
| User agent | Operator | Purpose |
|---|---|---|
| GPTBot | OpenAI | Model training |
| OAI-SearchBot | OpenAI | Search & answers |
| ChatGPT-User | OpenAI | User-requested fetch |
| ClaudeBot | Anthropic | Model training |
| Claude-SearchBot | Anthropic | Search & answers |
| Claude-User | Anthropic | User-requested fetch |
| PerplexityBot | Perplexity | Search & answers |
| Perplexity-User | Perplexity | User-requested fetch |
| Googlebot | Search & answers | |
| Google-Extended | Model training | |
| Bingbot | Microsoft | Search & answers |
| Applebot-Extended | Apple | Model training |
| CCBot | Common Crawl | Model training |
| Meta-ExternalAgent | Meta | Model training |
| Amazonbot | Amazon | Search & answers |
| DuckAssistBot | DuckDuckGo | Search & answers |
| Bytespider | ByteDance | Model training |
Example: allow AI search, block training
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: *
Allow: /
Sitemap: https://yourbrand.com/sitemap.xmlRules are evaluated the way major crawlers do it: the most specific user-agent group applies, and within it the longest matching path wins, with Allow winning ties.
Frequently asked questions
Will blocking GPTBot remove me from ChatGPT answers?
Blocking GPTBot opts your content out of model training. ChatGPT search uses OAI-SearchBot to index pages for citations, so you can block one and allow the other.
What happens if I have no robots.txt?
All crawlers are allowed. That is fine for visibility; add a robots.txt mainly to point to your sitemap or to block specific bots.
Do all bots respect robots.txt?
Reputable AI companies document their crawlers and state that they respect robots.txt. Some user-initiated fetchers may behave differently; check each provider's documentation.
Does this check my whole site?
It evaluates the rules for the exact path you enter (your homepage if you enter a domain). Rules can differ by path, so check important sections separately.
Readiness is step one. Visibility is the goal.
Track whether ChatGPT, Gemini, Perplexity and Claude actually mention you. Free monthly credits, no card.