AI crawler access checker
Check robots.txt rules for AI search and training crawlers. See which rule applies to your page.
How the check works
Fetch the origin’s robots.txt and evaluate the supplied path against each listed crawler’s product token. Show the matching rule and separate search crawlers from training crawlers.
- Robots.txt permission does not prove that a crawler can reach your page. Firewalls, authentication, and crawler policies can also affect access.
- This check does not measure citations, rankings, or whether content was used for training.
Read the result, then take action.
Illustrative example — not a live result
OAI-SearchBot is blocked
The page matches Disallow: / in the OAI-SearchBot group.
Next step: If you want this crawler to access the page, review the rule with your site administrator.
Common questions
Does allowing a crawler guarantee an AI citation?
No. A robots.txt rule only describes crawling permission. A search service decides whether to fetch, index, or cite the content.
Can I allow search while blocking training?
Some providers publish separate crawler identities for search and training. This report lists them separately so you can review each purpose.
Use this tool with an agent
Open this page in a WebMCP-compatible browser. Ask your agent to check your URL. The result appears here and is returned to the agent as JSON. Your agent can also use the public POST endpoint.
check_ai_crawler_access({ "url": "https://example.com" })
POST /api/free-tools/ai-crawler-checker
Content-Type: application/json
{ "url": "https://example.com" }Results contain findings, evidence, suggested fixes, a check time, and coverage limits. View the tool catalog.