Robots.txt validator
Syntax errors, sitemap, and which bots may crawl: Google, Bing and the AI ones (ChatGPT, Perplexity, Claude, Gemini), telling search from training.
Want this on every page, every day, with an alert when it gets worse? Run the free site check · See plans
Related guides
AI crawlers in robots.txt: search, user access and training
GPTBot, ClaudeBot, OAI-SearchBot, Google-Extended: what each AI crawler does, whether it respects robots.txt, and how to block training but stay in search.
Crawling and indexing: robots.txt, sitemaps and HTTP status
Why Google doesn't index a page: robots.txt, noindex, canonicals, sitemaps, HTTP status, JavaScript and the Search Console reasons, with a checklist.
Partners
Partner benefits with one coupon.
WPLIGHT20% OFFGet the offer from WP Rocket (opens in a new tab)
25% OFFGet the offer from Perfmatters (opens in a new tab)
Partner links. We may earn a commission at no extra cost to you.