Should You Allow GPTBot, ClaudeBot and PerplexityBot to Crawl Your Site

Allow the AI search crawlers without exception, and treat the training crawlers as a separate, smaller decision. That is the short answer for a seed to Series B AI startup whose marketing site, docs and blog exist to be found. This guide is for the founder, growth lead or engineer who has been asked whether to block AI bots, or who suspects a block is the reason ChatGPT, Perplexity and Claude never cite the site. It covers what each crawler does, what blocking it costs, the robots.txt we recommend and how to confirm the result in your server logs.

Which AI crawlers exist and what does each one do?

OpenAI, Anthropic and Perplexity each document separate bots for training, search and user requests, and Google and Microsoft handle AI use through controls on their existing search crawlers. The table below is built from each company's own crawler documentation, read on October 7, 2026. The column that matters most is the last one, because it tells you what you give up.

Swipe sideways to see every column

User agent	Company	Job	Follows robots.txt	What blocking it does
GPTBot	OpenAI	Collects content that may be used to train foundation models	Yes	Signals your content should not be used for training
OAI-SearchBot	OpenAI	Builds the results shown in ChatGPT search	Yes	Your pages are not shown in ChatGPT search answers, though they can still appear as navigational links
ChatGPT-User	OpenAI	Visits a page when a user asks ChatGPT or a custom GPT	Rules may not apply, since a user started the request	Not used to decide what appears in search
ClaudeBot	Anthropic	Collects content for model training	Yes, including Crawl-delay	Excludes your future content from training datasets
Claude-SearchBot	Anthropic	Indexes content to improve Claude's search results	Yes	May reduce your visibility and accuracy in Claude's search results
Claude-User	Anthropic	Fetches a page when a Claude user asks a question	Yes	May reduce your visibility for user-directed web search
PerplexityBot	Perplexity	Surfaces and links websites in Perplexity results, not used for training	Yes	Removes the crawler that puts your pages in Perplexity's results
Perplexity-User	Perplexity	Fetches a page for a user's question	Generally ignores robots.txt	Little effect, since it is user-triggered
Google-Extended	Google	A robots.txt token for Gemini training and grounding, with no separate user agent	Yes, as a token	No effect on inclusion or ranking in Google Search
Bingbot	Microsoft	Crawls for Bing, whose index feeds ChatGPT search and Copilot	Yes	Drops you from Bing and from the index Copilot draws on

Sources for each row are OpenAI's crawler overview, Anthropic's crawler help article (updated April 2026), Perplexity's crawler documentation and Google's list of common crawlers. Microsoft has not documented a separate AI crawler. Its September 2023 Bing Webmaster Blog post uses meta tags instead: NOCACHE limits what was then Bing Chat to the URL, title and snippet, and NOARCHIVE keeps the page out of those answers entirely, while both pages stay in Bing search results.

Should you allow GPTBot and the other training crawlers?

For most AI startups, allowing GPTBot and ClaudeBot costs little and blocking them gains little. Blocking only opts your future pages out of training data. It does not remove you from ChatGPT search or Claude's search results, because those use different bots. The case for blocking is strong only when your content is the thing you sell, such as a paid dataset, proprietary research or a paywalled course.

The case for allowing is weaker than many guides claim, and you should know that before deciding. What a model says about your product when it answers without searching comes from its training data, so excluding your site leaves third-party pages to describe you. We have not seen reliable data showing that allowing training crawlers makes a model recommend a product more often. Treat it as a low-cost default, not a growth lever.

Those ratios explain why publishers block, and why the calculation is different for a startup. A news site pays for that crawling in bandwidth and gets few visitors back. A SaaS site with a few hundred pages pays almost nothing, and the visitors it wants come through the search crawlers, which is where the real trade-off sits.

What do you lose by blocking the AI search crawlers?

You lose citations in that engine, usually without any warning. OpenAI states that sites which disallow OAI-SearchBot will not be shown in ChatGPT search answers. Anthropic says disabling Claude-SearchBot may reduce your visibility and accuracy in Claude's search results. Perplexity describes PerplexityBot as the crawler designed to surface and link websites in its results, and recommends allowing it.

Google works differently. Blocking Google-Extended does not take you out of Google AI Overviews or Google AI Mode, because Google's AI features documentation says those draw on the normal Search index and are controlled with nosnippet, data-nosnippet, max-snippet and noindex. Those controls also limit your regular Google results, so for a startup that wants traffic there is no clean way to opt out of Google's AI answers alone. Copilot sits on Bing's index, so a blocked Bingbot hurts both Bing rankings and Copilot answers. This is one reason adding your site to Bing Webmaster Tools is worth doing even if you never look at Bing's numbers.

• Block OAI-SearchBot and ChatGPT search stops showing your pages.
• Block Claude-SearchBot or Claude-User and Claude's search answers lose access to you.
• Block PerplexityBot and Perplexity loses the crawler that puts you in its results.
• Block Bingbot and you lose Bing and the index Copilot draws on.
• Block Google-Extended and Google Search is unaffected, though Gemini grounding is.

Which CDN and firewall settings block AI crawlers by default?

Three edge settings cause most of the silent blocks we look for: a bot-blocking toggle chosen at sign-up, a firewall or bot-fight rule that challenges unknown user agents, and a managed robots.txt that adds rules to your own file. None of them shows up in your code, so check the dashboard as well as the file. The wider crawlability audit is in technical SEO issues that stop AI startup websites from being crawled.

Cloudflare is the first place to check, since it says it manages and protects traffic for 20% of the web. In its July 2025 announcement, it said every new domain is now asked at sign-up whether to allow AI crawlers, and that more than one million customers had already chosen its one-click block. Its managed robots.txt prepends rules aimed at training crawlers such as Google-Extended and Applebot-Extended to your file. Someone who ticked a box during a DNS migration may have decided your AI visibility for you.

How should you configure robots.txt for AI crawlers?

Give each AI crawler its own group, allow every search and user crawler, and set the training bots to whatever you decided. Separate groups matter because of how matching works. Google's robots.txt specification says a crawler follows only the group with the most specific user agent that matches it and ignores the rest, so a bot with its own group skips your User-agent: * rules entirely.

# Search and user crawlers: allow
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Bingbot
User-agent: Googlebot
Allow: /
Disallow: /app/

# Training crawlers: allow (change to Disallow: / if your content is the product)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Allow: /
Disallow: /app/

# Everyone else
User-agent: *
Disallow: /app/

Sitemap: https://www.example.com/sitemap.xml

The /app/ line stands in for your logged-in product. Repeat it in every group, because a bot that matches a named group never reads the * group. OpenAI says robots.txt changes take about 24 hours to reach its search systems, and Perplexity says up to 24 hours. Anthropic does not publish a timeline. An llms.txt file does not override any of this, a point we cover in does llms.txt help your site get cited.

Set and verify your AI crawler policy

How do you check server logs for AI bot activity?

Search your access logs for each crawler's user-agent token, then confirm the IP addresses against the ranges each company publishes, because anyone can fake a user agent. OpenAI publishes ranges for GPTBot, OAI-SearchBot and ChatGPT-User, Perplexity for PerplexityBot and Perplexity-User, and Anthropic at claude.com/crawling/bots.json. For Googlebot, Google documents a reverse and forward DNS check.

# Count requests and status codes per AI bot (nginx or Apache combined log)
grep -Eio 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|bingbot' access.log | sort | uniq -c

# Which status codes each search bot gets (403 or 429 means a block)
grep 'OAI-SearchBot' access.log | awk '{print $9}' | sort | uniq -c

# Which pages ChatGPT users asked about
grep 'ChatGPT-User' access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Read the results in three passes. First, any 403, 429 or challenge response to a search bot is a block to remove. Second, the pages the search bots fetch show what each engine has indexed, which is often the homepage and pricing page and little else. Third, ChatGPT-User and Claude-User requests are pages a real person's question led an assistant to open, so the URLs they hit are a rough list of what buyers ask about you. Do not look for Google-Extended in your logs, because Google says it has no separate user agent. To see the visits that follow, pair this with tracking AI search referral traffic in GA4.

When does blocking AI crawlers make sense?

Blocking makes sense when your content is the product, when crawl load costs you real money, or when a contract or regulation restricts how your content can be used. Even then you rarely need to block everything. Disallow the paid or regulated directories for all AI bots, keep the marketing pages open to the search crawlers, and use Crawl-delay with ClaudeBot if load is the only concern, which Anthropic supports.

Avoid blocking by IP address. Anthropic warns that an IP block may not guarantee an opt-out because it stops the crawler from reading your robots.txt at all, and user-triggered fetchers such as Perplexity-User generally ignore robots.txt anyway. If your site runs on a framework that renders content in the browser, crawler access is only half the problem, which we cover in technical SEO issues that stop AI startup websites from being crawled. Crawler access is one part of technical SEO for SaaS, alongside rendering, indexing and site architecture.