How Claude Finds and Cites Web Sources When Recommending Tools

Claude finds sources by deciding whether a question needs current information, searching the web when it does, and citing the pages it drew on. If you're a founder whose customers mention Claude on sales calls and you want to know why it does or doesn't name your product, this is how its search works and how to check where you stand. ChatGPT, Perplexity, Gemini, Google AI Overviews, Google AI Mode and Copilot each have their own guides. This one covers Claude only.

How Claude finds and cites web sources, step by step

Claude runs a search loop: it judges whether the prompt needs the web, runs one or more searches, reads the results and answers with citations. Anthropic's web search tool documentation describes the loop for developers, and its help article on web search describes the same behaviour in the Claude apps.

From a buyer's prompt to a cited page

The quotes are short by design. In the API, each web search citation carries cited_text of up to 150 characters from the source. That is about one sentence, which is the unit your pages compete in: a sentence that states what your product does and for whom, without leaning on the sentence before it, is the kind Claude can lift.

When does Claude answer from training instead of searching?

Claude answers from training when a request draws on stable knowledge, and searches when it depends on things that change. Anthropic's tool documentation lists the triggers for search: recent events, current prices and statistics, information about specific organizations, people or products that might have changed, and explicit requests to look something up. It lists established facts, creative writing, analysis of content already in the conversation and greetings as cases where Claude answers directly.

Tool recommendations sit mostly on the search side, because they are questions about specific products whose features and prices change. But not always. A prompt like "explain how AI meeting note takers work" reads as stable knowledge and may be answered from training, naming whichever tools were well known when the model was trained. In the newer Claude app experience there is no web search toggle and Claude searches the web when it helps, per Anthropic's help article; users can still add "search the web" to a prompt to make sure it does.

Training-based answers come from public web content Anthropic crawled. Its crawler documentation says ClaudeBot collects web content that could contribute to model training. If the web said little about your product before the training cutoff, Claude's unsearched answers will say little too.

Where does Claude's web search get its results?

Anthropic's Trust Center subprocessor list names Brave Search for web search across all products, alongside TurboPuffer, which is also listed under web search, and Anthropic runs Claude-SearchBot to index content for its search responses. Claude's image results come from Bing, according to the web search help article. Brave is the one named external search engine, so its crawling rules matter, but it is not the only system behind Claude's results.

Brave runs its own crawler with an unusual rule. Brave's crawler help page says the crawler does not advertise a separate user agent, and that if a domain or page is not crawlable by Googlebot, Brave's bot will not crawl it either. Brave says its crawler is partly powered by data from users who opt into its Web Discovery Project, and it offers a form to submit a URL for re-fetching. It has no webmaster console like Google Search Console or Bing Webmaster Tools.

• A Googlebot block is also a Brave block: robots.txt rules that close a section to Googlebot close it to Brave, the search engine Anthropic names, too.
• Robots.txt does not de-index: Brave says a page is delisted with a noindex directive, not robots.txt, and it needs to re-fetch the page to see the change.
• Ranking on Google helps indirectly: the same crawlable, linked, clearly written pages that rank on Google and Bing are the ones any independent index tends to find.

Which Anthropic crawlers should you allow?

Allow Claude-SearchBot and Claude-User if you want Claude to cite you, and decide on ClaudeBot separately, because it only affects training. Anthropic's crawler documentation describes three robots with different jobs, and blocking the wrong one costs visibility.

Swipe sideways to see every column

Crawler	What it does	If you block it
ClaudeBot	Collects web content that could contribute to model training	Your future content is excluded from training datasets
Claude-SearchBot	Analyzes content to improve the relevance and accuracy of search responses	Your content is not indexed for search optimization, which may reduce visibility in search answers
Claude-User	Fetches pages when a person asks Claude a question that needs them	Claude cannot retrieve your content for user-directed requests

Anthropic says opting out requires robots.txt, applied on every subdomain, and that blocking its IP addresses may not work reliably because it stops the crawler reading your robots.txt. It supports the non-standard Crawl-delay directive if you want to slow crawling rather than stop it. The trade-offs across every AI crawler are in should you allow GPTBot, ClaudeBot and PerplexityBot to crawl your site.

# Let Claude cite you, opt out of training
User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: ClaudeBot
Disallow: /

How does Claude treat brand sites and product documentation?

Anthropic documents no preference for or against vendor sites. What it documents are mechanics that reward pages a model can read in one pass: full-page fetching when a user shares a link, short quoted citations, and, for developers building on Claude, the option to restrict search to chosen domains.

Those mechanics matter most for documentation. A developer who pastes your docs URL into Claude gets an answer built from what Claude-User fetches, so docs that serve their content as HTML text, with one task per page, are the safest bet. Anthropic's web fetch tool documentation says it does not support websites dynamically rendered with JavaScript, so serve docs text in the HTML. The API's allowed_domains setting lets a company build a Claude-based assistant that searches only approved sites, such as its own docs and a few trusted references, which means an internal tool may see only the sources its builder chose. The broader docs playbook is in how to do SEO for an AI product documentation site.

How can founders check whether Claude recommends their product?

Run the questions your buyers ask, several times each, with search forced on and then off, and log which products are named and which pages are cited. Repetition matters because Claude's lists vary between runs.

A Claude visibility check in under an hour

The same routine works across engines, and the no-cost manual check shows how to keep the log. For the ChatGPT side of the comparison, see how ChatGPT decides which software products to recommend.

Is Claude worth optimizing for?

Claude is small as a traffic source and growing fastest. SE Ranking's June 2026 study of 101,574 websites found Claude sent 2.62% of AI referral traffic in 2026, but grew 320% year over year, with its share jumping 159% in March 2026 alone. For a tool sold to a narrow technical audience, a small channel can still carry real pipeline, so check your own referral data before deciding.

Most Claude work overlaps with what you already do: crawlable pages, clear entity facts and third-party mentions. The Claude-specific steps are allowing Claude-SearchBot and Claude-User, keeping Googlebot rules open so Brave can crawl, and testing searched and unsearched answers separately. If Claude already shows up in your referral data or sales calls, a Claude SEO agency can take on that specific work; if it does not, fold Claude into the checks you run for every engine.