Un site qui bloque GPTBot, ClaudeBot, Google-Extended ou PerplexityBot ne pourra jamais être cité dans les réponses génératives. Ces quatre robots alimentent respectivement ChatGPT, Claude, Google AI Overviews/Gemini et Perplexity. Attention aussi aux pare-feux type Cloudflare qui peuvent bloquer ces robots avant même que le serveur ne voie la requête.
Exemple :
| Crawler | User-Agent | Operator | Powers | Recommendation |
|---|---|---|---|---|
| GPTBot | GPTBot | OpenAI | ChatGPT, ChatGPT Search, API | Allow |
| ClaudeBot | ClaudeBot | Anthropic | Claude, Claude Search | Allow |
| Google-Extended | Google-Extended | AI Overviews, Gemini | Allow | |
| PerplexityBot | PerplexityBot | Perplexity AI | Perplexity Search | Allow |
| CCBot | CCBot | Common Crawl | Training datasets for multiple AI models | Consider blocking |
| Bytespider | Bytespider | ByteDance | TikTok AI, internal LLMs | Block |
Recommandation de la configuration (Allow AI Search Bots)
# robots.txt — AI Crawler Configuration (2026)
# Place at: yoursite.com/robots.txt
# Traditional search engines (keep existing rules)
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /
# ===== AI CRAWLERS =====
# OpenAI — Powers ChatGPT Search
User-agent: GPTBot
Allow: /
# Anthropic — Powers Claude
User-agent: ClaudeBot
Allow: /
# Google AI — Powers AI Overviews & Gemini
User-agent: Google-Extended
Allow: /
# Perplexity — Powers Perplexity Search
User-agent: PerplexityBot
Allow: /
# Block training-only crawlers
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /
# Sitemap
Sitemap: https://yoursite.com/sitemap.xml