Can ChatGPT Actually See Your Website? Check in Ten Minutes

A growing share of your potential customers never see a search results page. They ask ChatGPT, Perplexity, or Google's AI Overviews, get an answer, and move on. If the AI engines cannot read your site, you are not in those answers, and you will never get an error message telling you so. Your site just quietly stops existing for an entire channel.

The frustrating part is that most invisible sites are invisible by accident. A security default here, a robots rule written years ago there, a page builder that renders everything in JavaScript. Ten minutes of checking tells you exactly where you stand. Here is the full inspection, in order of how often each layer is the culprit.

Layer One: Your CDN May Be Blocking AI Before It Reaches You

If your site sits behind Cloudflare, start here, because this is the layer most owners have never looked at. Since July 1, 2025, Cloudflare has blocked known AI crawlers by default for every new domain added to its network, unless you explicitly allow them [1]. Cloudflare sits in front of roughly one fifth of the web, so this single default flipped a huge portion of the internet from "AI can read you" to "AI needs permission" [2].

And the rules just changed again. On July 1, 2026, Cloudflare split AI traffic into three separately controllable categories: Search (crawlers that index your content to answer questions and send referrals), Agent (bots acting live on a user's behalf, like a chat assistant fetching your page), and Training (crawlers collecting content to train models). On September 15, 2026, new defaults take effect: for new domains, Training and Agent bots are blocked by default on pages that display ads, while Search remains allowed [3].

What this means practically: if you want to appear in AI answers, you generally want Search allowed at minimum, and you should make a deliberate decision about Agent, because agent-style fetching is how assistants pull your page in real time when a user asks about you.

The check: log in to your Cloudflare dashboard, open your domain, and find AI Crawl Control (or the older Block AI Bots setting under Security). Look at what is currently blocked. If you never touched this setting, do not assume it is open.

Layer Two: Your robots.txt May Have Opted You Out Years Ago

Every AI crawler announces itself with a user agent name, and well-behaved ones respect your robots.txt. The names that matter most right now: GPTBot and OAI-SearchBot (OpenAI), ChatGPT-User (OpenAI's live fetcher), ClaudeBot and Claude-User (Anthropic), PerplexityBot and Perplexity-User (Perplexity), and Google-Extended, which controls whether Google may use your content for its AI models.

The check takes one minute. Open yourfamousdomain.com/robots.txt in a browser and read it. Or from a terminal:

# Fetch your robots.txt and show any rules mentioning AI crawlers.
# Replace example.com with your domain.
curl -s https://example.com/robots.txt | grep -i -B1 -A2 \
  -e "GPTBot" -e "OAI-SearchBot" -e "ChatGPT-User" \
  -e "ClaudeBot" -e "Claude-User" \
  -e "PerplexityBot" -e "Google-Extended"

# No output means no AI-specific rules exist, which usually means allowed,
# unless a blanket "User-agent: * / Disallow: /" rule appears above.
# Check for that too:
curl -s https://example.com/robots.txt | grep -A2 "User-agent: \*"

Two common accidents live here. The first is a blanket Disallow put in place during development and never removed. The second is a copy-pasted "block all AI" recipe added during the 2023 panic about training data, which now also blocks the search-style crawlers that would be citing you. If you blocked everything back then, that decision deserves a rethink now that citations drive discovery.

Layer Three: Your Content May Not Exist Until JavaScript Runs

This is the silent killer for modern sites. Most AI crawlers read the raw HTML your server sends and do not execute JavaScript. If your pages are built as a client-side application where the HTML arrives nearly empty and JavaScript assembles the content afterward, an AI crawler sees the empty shell. Sites built with AI app builders and some page-builder configurations are especially prone to this, because the page looks perfect to a human in a browser and blank to a bot.

The check: view your page the way a crawler does.

# Fetch your page as a plain client and check whether your actual
# content is present in the raw HTML. Pick a distinctive sentence
# from the visible page and search for it.
curl -s https://example.com/your-page/ | grep -c "a distinctive phrase from your page"

# Result of 1 or more: your content is server-rendered. AI crawlers can read it.
# Result of 0: your content only exists after JavaScript runs. To most AI
# crawlers, this page is empty.

The browser version of the same test: right-click the page, choose View Page Source (not Inspect), and search for a sentence you can see on screen. If it is not in the source, it is not in what most AI engines receive.

A page that looks perfect to humans and empty to crawlers is the most expensive kind of invisible.

Layer Four: Meta Robots and the Noindex You Forgot

A meta robots tag with noindex tells crawlers to leave the page out. These get set at the page level, the template level, or site-wide (WordPress has a single Search Engine Visibility checkbox that noindexes everything), and they get forgotten. It happens to careful people. It has happened to us.

# Check any page for robots directives in one command.
curl -s https://example.com/your-page/ | grep -i "robots"

# Look for: 
# Also check the HTTP header version, which overrides silently:
curl -sI https://example.com/your-page/ | grep -i "x-robots-tag"

Bonus Layer: llms.txt, the Welcome Mat

Where robots.txt says who may enter, llms.txt is an emerging convention that tells AI systems what matters on your site: a plain-text map of your key pages and what they cover. Adoption by AI providers is still uneven, so treat it as cheap insurance rather than a ranking lever. It costs a few minutes, and our free LLMS.txt Generator builds one from your sitemap.

The Ten-Minute Checklist

Passing all four layers means AI engines can read you. Whether they cite you is the next problem, and that one is about how your content is structured once they arrive. We keep a full guide to that in our documentation: Content Structure for AI Overviews. And if you would rather have the whole inspection run for you, our free AI Citability Scorer checks access and structure in one pass: AI Citability Scorer.