In short
Getting found by AI means making sure an AI system can reach your pages, read them without running JavaScript, and understand which ones matter. It is the technical layer underneath AI visibility, and it is the part most sites get wrong before they ever start worrying about their content.
Key points
- AI crawlers are separate from Googlebot, and blocking one does not block the other.
- If your content needs JavaScript to appear, assume most AI crawlers never see it.
- robots.txt controls access, sitemap.xml aids discovery, llms.txt expresses priority.
- Test what a crawler actually receives rather than trusting what your browser shows you.
What this section covers
Before an AI system can quote you, three things have to be true. It has to be allowed to fetch your page. It has to be able to read what comes back. And it has to be able to work out which of your pages is the one worth quoting.
That is the whole of this section. It is unglamorous, it is mostly a one-time fix, and skipping it makes everything in the later sections pointless.
Why it is separate from SEO
The instinct is to assume that a site which ranks in Google is already fine for AI. Often it is. But not always, and the gaps are specific:
- AI crawlers are not Googlebot. GPTBot, ClaudeBot and PerplexityBot are separate agents with separate robots.txt rules. A site can be perfectly open to Google and closed to all three, usually by accident, often via a security product nobody remembers configuring.
- Most AI crawlers do not run JavaScript. Google renders. Assume the others do not. A page whose content appears only after a script runs is, to them, an empty page.
- Popularity is not priority. Search engines infer which page matters from links and behaviour. An AI system fetching a handful of your pages has far less to go on.
The order to do this in
Work down the list. Each step is pointless until the one above it is true.
- Access. Confirm AI crawlers are not blocked, in robots.txt and at your CDN or firewall.
- Readability. Confirm your content is in the served HTML, not assembled by a script.
- Discovery. Confirm a current sitemap and internal links that reach every page.
- Priority. Optionally, state which pages matter most.
Most sites fail at step one or two, spend their effort on step four, and wonder why nothing changed.
FAQ
What does getting found by AI mean? It means an AI system can fetch your page, read its content without executing JavaScript, and identify which of your pages answers a given question. It is access, readability and discovery, which together form the technical layer beneath AI visibility.
Is an AI crawler the same as Googlebot? No. GPTBot, ClaudeBot, PerplexityBot and others are separate agents with their own robots.txt rules and their own capabilities. Allowing Googlebot does not allow them, and blocking one does not block the others.
Do I need to do anything technical to appear in AI answers? Usually very little, and usually once. Most sites need to confirm they are not accidentally blocking AI crawlers and that their content is present in the served HTML. Beyond that, the work is in the content itself.
Does JavaScript stop AI from reading my site? It can. Google renders JavaScript before indexing; most AI crawlers do not. If your main content only appears after a script runs, assume those crawlers see an empty page.
What order should I do this in? Access, then readability, then discovery, then priority. Each step is wasted effort until the one before it is true.
Pages in this section
- Pillar opened with the llms.txt page.