Technical SEO Issues That Stop AI Startup Websites From Being Crawled by Google and AI Bots

The technical issues that stop a site being crawled by Google and AI bots are usually few, specific and fixable in a sprint. This guide is for founders and growth leads at AI startups whose site was built fast on Next.js, Webflow or Framer, often by a product engineer, and who see “Discovered, currently not indexed” in Google Search Console or never see their pages cited in AI answers. It lists the problems we find most often, explains how AI crawlers differ from Googlebot, and ends with an audit you can run in an afternoon.

Which technical issues stop a site being crawled by Google and AI bots?

Six issues account for most of it. They are copy rendered only in the browser, AI crawlers blocked without anyone deciding to block them, duplicate staging and app domains, slow or failing responses, broken URLs after a redesign, and docs that cannot be indexed. The table gives the symptom and the fix for each.

Swipe sideways to see every column

Issue	Typical cause	How to spot it	Fix
Copy only in JavaScript	Client-rendered pages, content fetched in the browser	Raw HTML from curl is missing your H1 and body copy	Server-render or pre-render marketing and docs pages
AI crawlers blocked	Copied robots.txt, CDN or firewall bot rules	Bot user agents get 403s or are disallowed	Allow the search crawlers you want, decide on training crawlers separately
Duplicate domains	Staging, preview and app subdomains left indexable	Your homepage title appears under another host in search	noindex staging, redirect duplicates, canonical to the main host
Slow or failing pages	Heavy scripts, cold starts, server errors	Core Web Vitals failures and 5xx errors in logs	Cut JavaScript, cache pages, fix errors
Broken URLs	Redesigns and docs moves without redirects	404s for bot requests in server logs	Permanent redirects from every old URL
Docs not indexable	Separate docs host, hash routes, login walls	Docs pages missing from Search Console and Bing Webmaster Tools	Own robots.txt and sitemap for the docs host, path-based URLs

How do AI crawlers differ from Googlebot?

AI crawlers read the HTML your server returns and, in most cases, stop there. Googlebot does more. Google's JavaScript SEO guide describes three stages, crawling, a rendering queue where JavaScript runs, and indexing from the rendered HTML. AI crawlers also come in several types per company, and each type does a different job.

Swipe sideways to see every column

Company	Training crawler	Search crawler	User-request fetcher
OpenAI	GPTBot	OAI-SearchBot	ChatGPT-User
Perplexity	None listed	PerplexityBot	Perplexity-User
Anthropic	ClaudeBot	Claude-SearchBot	Claude-User
Google	Controlled with the Google-Extended token	Googlebot	Not covered in this guide

The search crawlers are the ones that decide whether you can be cited. OpenAI's crawler documentation says sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers, and that user-initiated ChatGPT-User visits may not follow robots.txt. Perplexity's guide says PerplexityBot surfaces and links websites and is not used to train foundation models. Bingbot matters for AI answers too, because ChatGPT search and Copilot draw on Bing's index.

Why does client-side rendering hide your site from AI bots?

Because the crawler receives an empty shell and never runs the script that fills it. A marketing site built as a single-page app, or a page that fetches its copy in the browser after load, returns HTML with a root div and a script tag. Googlebot may render it later. Most AI crawlers will not. Google's guide says server-side or pre-rendering is still a great idea because not all bots can run JavaScript, and Bing's guidance notes that bingbot finds it hard to process JavaScript at scale on every page.

Framework choice is not the problem on its own. In the Next.js App Router, layouts and pages are Server Components by default, which means their HTML is rendered on the server. Trouble starts when a team marks whole pages with use client and loads the copy in an effect, or when pricing tables, FAQs and feature lists come from a client-side fetch. In the Webflow and Framer sites we review, the copy is usually in the HTML already, and the problems sit elsewhere, in staging domains and duplicate CMS paths.

# What does an AI crawler receive?
curl -s -A "OAI-SearchBot" https://www.example.com/pricing | grep -o "<h1[^>]*>[^<]*"
# No H1 in the output means the heading is added by JavaScript.
# Repeat for the homepage, a product page, a blog post and a docs page.

Are you blocking AI bots without knowing it?

Often, yes. Blocks come from three places, and only the first is visible in your code. A robots.txt copied from a template that disallows every AI user agent. A CDN or firewall setting that challenges or blocks bots. And a hosting default nobody revisited after launch.

If your domain was added to Cloudflare after that date, check the AI crawler setting in the dashboard as well as your robots.txt. Then decide crawler by crawler. Blocking GPTBot or ClaudeBot signals that your content should not be used for training. Blocking OAI-SearchBot, PerplexityBot or Claude-SearchBot keeps you out of answers. Anthropic's crawler page says blocking Claude-SearchBot may reduce your site's visibility in user search results, and that settings apply per subdomain. OpenAI says robots.txt changes take about 24 hours to apply, and Perplexity says up to 24 hours. The trade-offs of each choice are covered in should you allow GPTBot, ClaudeBot and PerplexityBot to crawl your site.

How do duplicate app, staging and marketing domains split your signals?

They give Google and Bing several copies of the same pages to choose between, and sometimes the wrong copy wins. AI startups tend to run a marketing site, an app at app.example.com, a docs host, preview deployments for every branch and a staging site. Any of them can be crawled if it is reachable and not marked noindex.

• Staging. Webflow's own SEO course recommends disabling indexing of your staging subdomain so test versions do not appear in search. The same applies to any staging host.
• Preview deployments. Check that branch previews return a noindex header or sit behind authentication.
• The app. Login and signup pages can be indexed, but dashboards and user content should be noindexed or behind auth.
• Host variants. www and non-www, and http and https, should each redirect to one version.

Use the right tool for each case. Google's robots.txt introduction says robots.txt is not a mechanism for keeping a page out of Google, and that a disallowed page can still be indexed if other sites link to it. Use noindex for pages that should not appear. Use redirects or rel="canonical" for duplicates, which Google's canonicalization guide calls strong signals, while sitemap inclusion is a weak one.

Does page speed affect crawling by Google and AI bots?

Speed affects crawling through errors and timeouts more than through scores. Google's crawl budget guide says that when a site slows down or responds with 5xx errors or 429 rate-limit signals, Google crawls less. For users and for Google's page experience signals, web.dev's Core Web Vitals guidance sets the good thresholds at a Largest Contentful Paint within 2.5 seconds, an Interaction to Next Paint of 200 milliseconds or less and a Cumulative Layout Shift of 0.1 or less, measured at the 75th percentile of page loads.

Broken URLs cost more with AI crawlers than with Google. The Vercel analysis found that 34.82% of ChatGPT's fetches and 34.16% of Claude's hit 404 pages, against 8.22% for Googlebot. After a redesign, keep a redirect map from every old URL, including old blog and docs paths.

Crawl budget is rarely the issue unless Search Console shows many URLs as “Discovered, currently not indexed”. Google's crawl budget guide is written for sites with more than a million unique pages that change weekly, or more than 10,000 pages that change daily, or for sites where a large share of URLs sit in “Discovered, currently not indexed”. If your pages are crawled the day you publish them, Google says you do not need it.

Why do docs and programmatic pages fail to get indexed?

Docs fail mostly for crawl reasons, programmatic pages mostly for quality reasons. Docs often sit on a separate host with its own robots.txt, use hash-based routes that give crawlers one URL for the whole reference, or render API references in the browser. The fixes are in the guide to SEO for an AI product documentation site. Programmatic pages are usually crawled fine and then left out of the index because they add little beyond a template. Search Console reports that as “Crawled, currently not indexed”, and a technical fix alone rarely changes it. The page has to earn its place, which is the subject of when programmatic SEO makes sense for an AI SaaS product.

How do you audit whether your site is crawled by Google and AI bots?

Run these checks in order. The first three catch most of the problems we see, and none of them needs a paid tool. For the full scope of a technical audit across the marketing site, app and docs, see technical SEO for SaaS.

The afternoon crawl audit

Which technical fixes will not help?

Some popular fixes do nothing for crawling. An llms.txt file does not make JavaScript-rendered pages readable, and Google's AI features documentation says no AI text files or special markup are needed to appear in Google AI Overviews or Google AI Mode. The evidence on whether it helps elsewhere is in does llms.txt help your site get cited by AI engines. Schema markup helps engines understand a page they can already read, but it cannot replace missing HTML, as the guide to which schema markup helps AI search engines understand your product explains. Resubmitting sitemaps does not lift a bot block. Crawl budget tuning on a site with a few hundred pages is time taken from the fixes above.