SEO 工具
Robots.txt 测试器
免费 robots.txt 测试器，支持 Googlebot、GPTBot、ClaudeBot、PerplexityBot 及所有主流爬虫。粘贴 robots.txt，选择 user-agent，同时测试多个 URL 路径——每条结果显示路径是允许还是被屏蔽，以及触发匹配的具体规则，完整复现搜索引擎和 AI 爬虫使用的 RFC 9309 匹配算法。
主要用途
This tester recreates the RFC 9309 robots.txt matching algorithm that search engines and major AI crawlers follow, so you can confirm exactly which URLs a given user-agent may fetch. Paste your robots.txt contents, choose a crawler such as Googlebot, GPTBot, ClaudeBot, PerplexityBot, or Google-Extended, then list the URL paths you want to check. The tool reports allowed or blocked for each path and, crucially, names the specific directive responsible — including how longest-match and allow-over-disallow precedence rules resolve conflicts. It fills the gap left when Google retired its official robots.txt tester in 2023, and it is especially useful now that publishers want to control AI training crawlers as deliberately as they manage traditional search bots.
所在链路
This tester sits in the technical SEO setup stage, verifying crawl directives before they ship and after every robots.txt edit.
核心功能
- RFC 9309-compliant matching that mirrors real crawler behavior
- Support for Googlebot plus AI crawlers like GPTBot and ClaudeBot
- Per-path allowed or blocked verdict for batch URL testing
- Identification of the exact directive controlling each result
- Correct handling of longest-match and allow-over-disallow precedence
适合谁用
- Technical SEOs validating crawl rules before deploying robots.txt
- Publishers deciding which AI training crawlers to allow or block
- Developers debugging why a page is or is not being indexed
新手提示
- Robots.txt controls crawling, not indexing; a blocked URL can still appear in results, so use noindex for true removal.
- Test the exact user-agent string a crawler sends, because rules for Googlebot do not automatically apply to AI bots like GPTBot.
- Remember that the most specific matching rule wins, so a broad Disallow can be overridden by a narrower Allow.