AI Crawler Access Agent Skill
Check which AI crawlers can reach your site across robots.txt, meta tags, and HTTP headers, and fix the blocks costing you AI visibility.
/geo-crawlersThe AI Crawler Access Agent Skill maps which AI crawlers your site allows and which it blocks, across robots.txt, robots meta tags, and HTTP headers.
Crawler access is the foundational requirement for AI visibility. If GPTBot, ClaudeBot, PerplexityBot, or Google-Extended cannot fetch a page, no amount of content quality will get it cited — the content is simply not available to the model.
What this skill helps your agent do
- Build a complete access map: which named AI crawlers are allowed, blocked, or unaddressed.
- Catch blocks you did not intend, including blanket rules and inherited defaults.
- Distinguish training crawlers from retrieval crawlers, so you can allow citation without allowing training.
- Recommend the specific
robots.txtlines to change.
When to use it
Run it first, before any other AI-visibility work, and again after any change to robots.txt or your CDN's bot rules.
What you get back
A per-crawler access table with the rule responsible for each verdict, plus the exact edits to make. Where a block looks deliberate, it says so rather than telling you to remove it — declining to be used for training is a legitimate choice.