Should You Opt Out of AI Training With the New Shared robots.txt Control
Quick answer
Should you opt out of AI training with the new shared robots.txt control? For most B2B SaaS companies, no. Google, Microsoft and OpenAI parse the training preference separately from the crawlers that feed their answers, so opting out does not remove you from ChatGPT, Perplexity, Claude, Google AI Overviews, Google AI Mode or Copilot. The one trap is Gemini, where a bare opt-out also removes grounding unless Google-Extended is explicitly allowed.
On this page
- Should you opt out of AI training with the new shared control?
- What did Google, Microsoft and OpenAI announce on October 30?
- What is the difference between AI training and AI grounding?
- Does opting out of AI training remove you from AI answers?
- What should a B2B SaaS set in robots.txt and on its pages?
- What breaks if you still rely on NoArchive or NoCache?
- When does opting out of AI training make sense?
- Who can help after the robots.txt decision is made?
Key takeaways
- Google, Microsoft and OpenAI said on October 30, 2026 that they will honor one shared robots.txt token for AI training, and nothing is live until 2027.
- Opting out of training does not affect citations in ChatGPT, Perplexity, Claude, Google AI Overviews, Google AI Mode or Copilot, according to each company's own documentation.
- Gemini is the exception, because Google reads the shared opt-out as a Google-Extended disallow, which also removes Gemini Apps grounding unless Google-Extended is explicitly allowed.
- Bing now separates training from grounding and stops reading NoArchive and NoCache as AI preferences after March 1, 2027, so restate any restriction you meant to keep.
- Make the training decision with legal and keep every search and grounding crawler allowed; Ahrefs' May 2025 data shows under 6% of sites block GPTBot.
- SearchAxe does the crawler access and page work that follows this decision as part of its GEO agency engagements, measured against a baseline before anything changes.
You should opt out of AI training only when you have a reason to keep your content out of model training, because the training choice no longer decides whether you appear in AI answers. On October 30, 2026, Google, Microsoft and OpenAI each said they will honor one shared robots.txt token for training, and Microsoft split training from grounding so a site can refuse one and allow the other. If you run marketing or growth for a business and legal has asked whether to block AI training, this article explains what changed, what it leaves alone, and the settings that keep you in ChatGPT, Perplexity, Claude, Gemini, Google AI Overviews, Google AI Mode and Copilot whatever you decide. Which individual crawlers to allow is covered separately in our guide to GPTBot, ClaudeBot and PerplexityBot.
Should you opt out of AI training with the new shared control?
Most B2B SaaS companies should not, and the ones with a real reason to can now do it without losing search or AI answer visibility at Google, Bing or OpenAI. The decision is commercial and legal, not a visibility decision, because all three companies parse the training preference separately from the crawlers and preferences that feed their answers. OpenAI's crawler documentation says each setting is independent of the others and gives the example of allowing OAI-SearchBot for search while disallowing GPTBot for training. Google's Search Central announcement says the shared token will be read as a Google-Extended disallow, and Google's crawler documentation says Google-Extended does not affect inclusion or ranking in Google Search. Microsoft's Bing announcement says its training and grounding preferences do not change eligibility for Bing search results.
Most SaaS companies should leave training allowed because the product pages they publish are the pages they want a model to have read. When an assistant answers a category question from memory rather than from a live search, your product can only be in that answer if descriptions of it were in the training data. Opting out closes that path for future models with no offsetting gain for a company whose content exists to be found. The case for opting out is stronger when the content itself is the product, covered at the end of this article.
What did Google, Microsoft and OpenAI announce on October 30?
Three posts on the same day, all backing the same robots.txt token, with different timing and different extras. The token comes from draft-kelley-aipref-training-preference-signal-00, an individual Internet-Draft by Kevin Kelley of Anthropic, dated October 30, 2026, that the IETF has not endorsed. Google's post, by John Mueller, says Google aims to support the control in 2027, and its common crawlers page says early 2027. Microsoft's post says Bing begins honoring the preference as it crawls on February 1, 2027. Search Engine Journal's November 1 report says OpenAI aims to honor the signal within the next few months while its current robots.txt policy stays in effect, and that Anthropic has started development but has not published a post.
| Company | What it committed to | When | What it added beyond the token |
|---|---|---|---|
| Reads a token disallow as a Google-Extended disallow when Google-Extended is not named | Early 2027 | Nothing new. Google-Extended and the preview controls keep working | |
| Microsoft | Honors the token for a site or path | February 1, 2027 | Page-level training override, a separate page-level AI Grounding preference, retirement of NoArchive and NoCache |
| OpenAI | Will honor the token | Within a few months, per Search Engine Journal | Nothing. GPTBot and OAI-SearchBot stay independent |
| Anthropic | Wrote the draft, development under way | No date | ClaudeBot, Claude-User and Claude-SearchBot unchanged |
Nothing is live today. Microsoft's FAQ answers "Does anything change today?" with "No", and Google's timeline is next year, so a line you add this week is a statement of intent until 2027.
Find out which AI engines recommend you
A free consultation on where you appear today across AI engines and search.
Request a Free ConsultationWhat is the difference between AI training and AI grounding?
Training changes what a model knows. Grounding changes what a model reads before it answers you. Microsoft's Bing Webmaster Tools FAQ defines AI training as using an asset to modify the learned parameters of a generative AI model, and AI grounding as using content as input to help inform AI-generated responses, with links and citations providing a path back to the page. The same FAQ says that indexing and ranking pages in traditional search results is not grounding. For a company that wants to be recommended, grounding is the one that produces citations, so it is the one you protect.
Until now the two were tangled. Bing read NoArchive and NoCache as one decision covering both, which is why Microsoft says it built the new preferences. Google-Extended, by Google's crawler documentation, still covers both uses at once, training of Gemini models and grounding in Gemini Apps and on Vertex AI, which is why the shared token has a side effect at Google that it does not have at Bing. Gary Illyes made the underlying point in a Search Central video reported by Search Engine Journal in October 2026. Robots.txt is an access preference protocol, and what a company does with content after crawling it is governed by that company's documentation, not by the file.
Does opting out of AI training remove you from AI answers?
Not at ChatGPT, Perplexity or Claude, not in Google AI Overviews or Google AI Mode, and not at Copilot. The one place it can is Gemini, because Google plans to treat the shared opt-out as a Google-Extended disallow, and Google-Extended governs grounding in Gemini Apps as well as training. Google's announcement states the flip side directly, that websites which specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps. The table lists what each engine has published.
| Engine | Training control | What decides whether you are cited | Effect of a training opt-out on answers |
|---|---|---|---|
| ChatGPT | GPTBot, then the shared token | OAI-SearchBot access, independent of GPTBot per OpenAI | None |
| Perplexity | None needed. PerplexityBot is not used for foundation model training, per Perplexity | PerplexityBot access | None |
| Claude | ClaudeBot | Claude-SearchBot and Claude-User access | None. Blocking ClaudeBot does not affect search retrieval, per Anthropic |
| Gemini | Google-Extended, and the shared token read as Google-Extended | Google-Extended, which also covers grounding in Gemini Apps | Loses Gemini Apps grounding unless Google-Extended is explicitly allowed |
| Google AI Overviews | Not applicable | Googlebot access and the nosnippet, data-nosnippet and max-snippet controls | None. Google says there are no requirements beyond being indexed and snippet-eligible |
| Google AI Mode | Not applicable | Same as Google AI Overviews | None |
| Copilot | Shared token or Bing page-level ai-training tag | Bing's page-level AI Grounding preference, default Allow | None. Training and grounding are independent, per Microsoft |
The Gemini row is the trap. A site owner who adds the shared token to stop training will, under Google's stated plan, also stop Gemini from grounding answers on the site, unless the file also contains an explicit "User-agent: Google-Extended" group with "Allow: /". Google says specific controls keep working alongside the joint signal, and the IETF draft says specific expressions should take precedence over general ones. Bing has the opposite trap. The token does nothing to grounding, but a page-level "ai-grounding:disallow" tag does, and Microsoft says Disallow and DisallowExtended will significantly reduce visibility in its AI experiences. For Google AI Overviews and Google AI Mode, Google's AI features documentation points to the preview controls, not a training token.
Those figures predate the shared token and do not separate SaaS sites from publishers, but they show that opting out of training is a minority position, and a B2B company copying a publisher's robots.txt is copying a decision made for a different business model.
What should a B2B SaaS set in robots.txt and on its pages?
Allow every search and grounding path, decide training on its own, and write the decision in a way each company will read the way you intend. The order below is the one we use, with the reason each step sits where it does.
- Confirm Googlebot, Bingbot, OAI-SearchBot, PerplexityBot and Claude-SearchBot are allowed.These crawlers feed the indexes answers are built from, and OpenAI says a site opted out of OAI-SearchBot will not be shown in ChatGPT search answers.
- Make the training decision with legal, in writing, before touching robots.txt.It is a licensing question, and one line will soon express it to several providers at once, so it should not be set by whoever edits the file.
- If you opt out, add the token group and an explicit Google-Extended rule.Google reads a bare opt-out as a Google-Extended disallow, which removes Gemini Apps grounding, and an explicit Google-Extended group overrides that.
- Leave Bing's AI Grounding preference at its default on every page you want cited.Allow needs no tag, and a page-level Disallow or DisallowExtended is what Microsoft says reduces visibility in its AI experiences.
- Search your templates and headers for noarchive and nocache before December 15, 2026.Bing stops reading them as AI preferences after March 1, 2027, so any restriction you meant to keep must be restated.
- Check server logs after February 1, 2027.That is when Bing starts honoring the new preferences, and a log line is the only proof a crawler still fetches the pages you want cited.
For a company that opts out of training and still wants every answer engine to cite it, the robots.txt groups look like this. The Google-Extended group is the line most teams forget.
User-agent: __usage_ai-training
Disallow: /
User-agent: Google-Extended
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Allow: / The GPTBot and ClaudeBot groups are there because nobody honors the shared token yet, and Search Engine Journal reports that OpenAI's current robots.txt policy stays in effect until it does. If you want to keep Gemini training out as well, replace the Google-Extended Allow with a Disallow and accept the loss of Gemini Apps grounding, because Google has no middle setting today. Bing's grounding choice is made per page with a meta tag or header, and for a page you want cited the right action is to add nothing.
What breaks if you still rely on NoArchive or NoCache?
After March 1, 2027, Bing stops reading those tags as training or grounding preferences, and a page that carried them without a replacement defaults to Allow for grounding until Microsoft recrawls it. Microsoft's FAQ sets out four dates. On December 15, 2026, a temporary grounding preference opens in Bing Webmaster Tools, only for sites using NoArchive or NoCache as of that date. On February 1, 2027, Bing begins honoring the new preferences as it crawls. March 1, 2027 is the deadline to publish the new preferences and record the temporary one. On September 30, 2027, temporary preferences expire. Microsoft also says that once it detects new AI preferences on a site, it disregards any remaining NoArchive and NoCache tags there.
For most SaaS sites this is good news that needs no action, since the tags were never meant to limit AI use. The sites that must act are the ones that added the tags on purpose to keep documentation or gated research out of Bing's AI answers, because that restriction lapses silently unless they publish the page-level grounding tag and record the temporary preference. The reverse mistake is more common in our experience, a tag added years ago for an unrelated reason that is quietly limiting AI visibility today and that the new rules will clear by default.
When does opting out of AI training make sense?
When the content is the thing you sell, when a licensing deal depends on it, or when a specific path holds data you do not want reproduced. A media company, a research firm or a benchmark publisher loses something real when a model learns its material, and the token gives them one line to say so across providers. A SaaS company with a proprietary dataset or a gated report can opt out for /research/ alone while leaving product pages open, which the IETF draft supports through ordinary robots.txt path matching. None of these cases requires giving up citations, provided Google-Extended and Bing grounding are handled as above.
What opting out does not do is protect your positioning. Training opt-outs apply to future models, and Microsoft says neither choice guarantees inclusion, traffic or payment. If an assistant describes your product wrongly today, the fix is the pages and third-party sources it reads at answer time, the process in how AI search engines decide which sources to cite, and for the Bing-backed engines it starts with Bing Webmaster Tools. A team that owns its robots.txt and templates can make every change in this article in an afternoon, and the in-house versus agency trade-offs apply to what comes after. Where a generative engine optimization agency such as SearchAxe earns its place is in the months that follow, when the question stops being what the crawlers may do and becomes which of your pages they actually quote.
Who can help after the robots.txt decision is made?
SearchAxe is an SEO, AEO and GEO agency that picks up where this article stops. It checks that Googlebot, Bingbot, OAI-SearchBot, PerplexityBot and Claude-SearchBot are allowed, searches templates and headers for noarchive, nocache and nosnippet tags nobody can explain, and writes the robots.txt groups, including the explicit Google-Extended rule, once legal has made the training decision. The work then moves to the pages each engine reads and quotes. It does this for businesses across 16 industries, from SaaS and AI startups to finance and healthcare firms. Every engagement starts with a baseline recorded in the first 30 days across ChatGPT, Perplexity, Claude, Gemini, Google AI Overviews, Google AI Mode and Copilot plus Google and Bing, reported monthly as citations, rankings and the leads they drive. The GEO agency page sets out the workstreams. The first step is a free consultation.
Frequently asked questions
Does the __usage_ai-training token work today?
No. Google says it aims to support the control in 2027 and its crawler documentation says early 2027. Microsoft begins honoring it on February 1, 2027. OpenAI aims to honor it within the next few months, per Search Engine Journal, and until then its GPTBot rule remains the control. Adding the line now is harmless, but keep the provider-specific groups you already have, because those are what each company reads today.
Does blocking GPTBot keep my site out of ChatGPT search?
No. OpenAI's crawler documentation says GPTBot is used for training foundation models, OAI-SearchBot is used to surface websites in ChatGPT's search features, and each setting is independent of the others. A site can disallow GPTBot and allow OAI-SearchBot and still appear in ChatGPT search answers. The setting that removes you is disallowing OAI-SearchBot, and OpenAI says it takes about 24 hours for a robots.txt change to reach its search systems.
Will opting out of AI training affect my Google rankings?
No. Google's crawler documentation says Google-Extended does not impact a site's inclusion in Google Search and is not used as a ranking signal, and Google plans to read the shared token as a Google-Extended disallow. Google AI Overviews and Google AI Mode are also unaffected, because Google says the only requirement for them is being indexed and eligible to show with a snippet. The one Google product that is affected is grounding in Gemini Apps, which Google-Extended also governs.
What happens to noarchive and nocache tags after March 1, 2027?
Bing stops reading them as AI training or AI grounding preferences. Microsoft says the new preferences replace them for those uses, and once it detects new preferences on a site it disregards any remaining NoArchive and NoCache tags there. Sites that used the tags on purpose must publish the page-level grounding preference and, if they used the tags as of December 15, 2026, record a temporary preference in Bing Webmaster Tools before March 1, 2027.
Can I opt out of AI training and still be cited by Gemini?
Yes, but only with an explicit rule. Google plans to treat a shared opt-out as a Google-Extended disallow when Google-Extended is not specified, and Google-Extended governs grounding in Gemini Apps as well as training. Google's announcement says company-specific controls keep working alongside the token, so a robots.txt that disallows __usage_ai-training and separately allows Google-Extended keeps Gemini grounding while still telling other providers not to train. The trade-off is that Google may then train Gemini on the site.
Who can help decide which AI crawlers to allow and what to fix next?
SearchAxe, an SEO, AEO and GEO agency, reviews robots.txt, CDN rules and page-level tags for businesses across 16 industries, including SaaS and fintech, and confirms every search and grounding crawler can reach the pages you want cited. The training choice stays with your legal team. The pages engines quote are then rewritten and tracked monthly against a baseline across ChatGPT, Perplexity, Claude, Gemini, Google AI Overviews, Google AI Mode and Copilot. Details are on the GEO agency page.