AI Crawler Telemetry & Bot Logs
Monitor, verify, and govern search crawlers from OpenAI, Anthropic, Google, and Perplexity in real-time.
AI Crawler Telemetry & Bot Logs
As AI search engines index the web, they dispatch dedicated crawler user-agents that differ from standard Googlebot or Bingbot.
Zoventic GEO includes an On-Premises AI Crawler Telemetry Engine that identifies, logs, and verifies genuine AI crawler hits with zero external tracking dependencies.
1. Supported AI Search Crawlers
Zoventic GEO actively recognizes and parses headers for:
| Crawler User-Agent | Parent Organization | Primary Purpose |
|---|---|---|
| GPTBot | OpenAI | ChatGPT Search & AI browsing indexation |
| PerplexityBot | Perplexity AI | Real-time answer engine citation search |
| ClaudeBot | Anthropic | Claude Search & Assistant knowledge indexation |
| Google-Extended | Gemini training & AI Overviews ingestion | |
| Applebot-Extended | Apple | Apple Intelligence & Siri search feeds |
| Amazonbot | Amazon | Alexa & Rufus eCommerce recommendation |
| ByteSpider | ByteDance | Search indexing & product feeds |
2. Forward-Confirmed Reverse DNS (FCrDNS) Verification
Malicious scraping bots frequently fake their User-Agent strings (claiming to be GPTBot or ClaudeBot) to bypass site firewalls.
Zoventic GEO performs FCrDNS (Forward-Confirmed Reverse DNS) validation:
- Checks the incoming IP’s reverse DNS hostname (e.g.
crawl-***.openai.comor***.anthropic.com). - Confirms that forward DNS of that hostname resolves back to the authentic IP address.
- Tags verified crawlers with a green “Verified Bot” badge, and flags impostor scrapers in red.
3. Crawler Governance & Robots.txt Customization
In Settings > AI Search Bot Permissions, merchants can selectively toggle crawl permissions for any bot vendor.
Zoventic GEO dynamically handles this by generating valid directives in your WordPress virtual robots.txt:
User-agent: GPTBot
Allow: /llms.txt
Allow: /llms-full.txt
Allow: /product/
User-agent: ByteSpider
Disallow: /