AI crawler checker
See which AI crawlers can read any website, and exactly which robots.txt rule allows or blocks them.
Check AI crawler access in three steps
- STEP 1
Enter a website
Type any domain. We read its public robots.txt, llms.txt and home page tags.
- STEP 2
See every bot
Each AI crawler is marked Allowed, Blocked or Partly blocked, with the exact rule and line.
- STEP 3
Fix it
Open the result in our robots.txt generator to change the rules in a click.
Why check AI crawlers?
AI answer engines like ChatGPT, Claude, Perplexity and Gemini can only cite pages their crawlers are allowed to read. Many sites block them by accident, for example with an old robots.txt template or a security plugin. Others want to block training while staying visible in AI search. This checker shows the real situation in seconds.
GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI), ClaudeBot, Claude-SearchBot and Claude-User (Anthropic), Google-Extended and Gemini-Deep-Research (Google), PerplexityBot and Perplexity-User, Applebot-Extended, meta-externalagent, Meta-WebIndexer, Amazonbot, Bytespider, CCBot, DuckAssistBot, MistralAI-User, cohere-ai and Diffbot, plus Googlebot and Bingbot.
The bot can read the home page, but some folders are blocked for it, for example /blog/ or /admin/.
No. Without robots.txt every crawler is allowed. Adding one lets you block private folders and list your sitemap.
No tool can see inside AI companies. This checks whether their crawlers are allowed. To see real visits, look for these bot names in your server logs or CDN analytics.
A proposed file (yoursite.com/llms.txt) that gives AI tools a short, clean summary of your site and links to your key pages. Our llms.txt generator can make one for you.
Related free tools
People also search for
- AI crawler checker
- is GPTBot blocked
- check ClaudeBot robots.txt
- block AI crawlers
- AI bot robots.txt checker
- PerplexityBot checker
- Google-Extended robots.txt
- check if AI can crawl my site
- GEO audit
Questions about the AI crawler checker?
Stuck, found a bug, or have an idea to make it better? Send a message and we'll reply within 1–2 working days.
- Email us at pketu916@gmail.com
- Saw this tool used to deceive or harm someone? Choose Report misuse as the topic.
- Read the Terms of Use and Privacy notice for this tool.