Should I let AI crawlers like GPTBot see my website?
For most businesses that want customers, yes: at least let the AI search crawlers in, because blocking them can keep you out of AI answers. GPTBot is different: OpenAI says it collects content for training, and you can block it without leaving ChatGPT search.
ShoaibUpdated 3 min read
Which bot does what?
| Bot | Company | What it's for, per the company's docs |
|---|---|---|
| GPTBot | OpenAI | Content that may be used to train its AI models |
| OAI-SearchBot | OpenAI | Surfacing websites in ChatGPT search results |
| ChatGPT-User | OpenAI | Visiting a page when a user asks; not automatic crawling |
| Google-Extended | A robots.txt token for Gemini training; doesn't affect Google Search | |
| ClaudeBot | Anthropic | Content that may be used for training |
| Claude-SearchBot | Anthropic | Improving Claude's search results |
OpenAI notes that for ChatGPT-User, because a person started the visit, "robots.txt rules may not apply."
What's the trade-off?
- Block training bots (GPTBot, ClaudeBot, Google-Extended) if you don't want your content used to train AI models. OpenAI and Google both say this is separate from search visibility.
- Allow search bots (OAI-SearchBot, Claude-SearchBot) if you want to be found and cited in AI answers. OpenAI recommends allowing OAI-SearchBot for this.
A common middle path in robots.txt:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
What should I check first?
Website platforms and security tools can be set to block AI bots, so look at your live yoursite.com/robots.txt before deciding. Then read how to rank in ChatGPT search, what llms.txt is, and, since AI answers aren't your only channel, what search engine marketing (SEM) is.
Want your site set up for AI search as part of your digital marketing? See our SEO, GEO and AEO service.