Does Blocking GPTBot Cost You Citations? Two Studies Disagree... Both Are Right.

Two serious studies asked whether blocking AI crawlers reduces how often AI engines cite you. One found that blocking barely matters. The other found that blocking is close to a delisting. Both are sound, and the gap between them is the most useful thing either of them produced.

What the two studies found

BuzzStream, working with Citation Labs' tracking tool across 4 million citations from 3,600 prompts spanning ChatGPT, Gemini, AI Overviews and AI Mode, looked at the top 50 news sites that block OpenAI's crawlers. Among sites blocking GPTBot, 88.2% still appeared in the citation data. Among those blocking OAI-SearchBot, 82.4% still appeared. Among those blocking ChatGPT-User, 70.6% still appeared. One site, cnbc.com, blocked all three and turned up 1,298 times.^[1]

Cloro cross-referenced crawler rules against citation counts for 1,058 prominent domains and normalized the result by how often each domain already appears in Google organic results, which controls for the obvious objection that cited domains are simply more prominent. The median GPTBot-blocking domain earned 0.003 ChatGPT citations per Google ranking. The median domain allowing GPTBot earned 0.417.^[2]

Those are not small differences in emphasis. One says blocking is nearly costless. The other says blocking is a two-order-of-magnitude penalty.

Why they disagree

The populations are different in a way that decides the answer. BuzzStream measured the top 50 news publishers. Those domains are among the most linked, most syndicated and most licensed properties on the web. Several have content agreements that supply AI firms directly, bypassing crawling entirely. Their reporting is quoted, aggregated and rewritten everywhere else, so an engine can cite the story through a hundred other doors. And their robots.txt directives are frequently not enforced at the server, so the crawl happens regardless.

Cloro measured ordinary prominent domains and asked a different question: given how visible you already are in search, how often do you get cited? For a site without a licensing deal and without a thousand syndication partners, robots.txt is the whole gate.

The detail that settles it is inside Cloro's own data. The damage follows the crawler's owner. Domains blocking PerplexityBot showed a median Perplexity citation propensity of zero, against 1.167 for the rest. Blocking a rival provider's bot barely moved the other engine's numbers.^[2] That is not the signature of a confound. That is the signature of a mechanism.

If you are the New York Times, blocking costs you little. If you are a normal business, blocking is a delisting.

Training bots, retrieval bots, and the one you cannot opt out of

Most of the confusion in this debate comes from treating "AI bots" as one thing. Providers separate them, and the separation is the entire point. OpenAI documents three roles: GPTBot for model training, OAI-SearchBot for inclusion in ChatGPT search answers, and ChatGPT-User for fetches triggered by a person in a session. The rules are independent, so blocking GPTBot does not block ChatGPT search.^[6] Anthropic separates ClaudeBot, Claude-SearchBot and Claude-User. Perplexity separates PerplexityBot from Perplexity-User.^[7]

That last pair is the trap. Google-Extended governs certain non-Search generative uses and does not control whether you appear in Google Search.^[7] AI Overviews and AI Mode are built on the Search index that Googlebot populates. There is no directive that removes you from AI Overviews while leaving your organic rankings intact. Opting out of Google's AI surfaces means opting out of Google.

BuzzStream's survey of 100 top news sites found 79% blocking at least one training bot and 71% also blocking at least one retrieval bot.^[3] The second number is where publishers are hurting themselves. Blocking training is a defensible position. Blocking retrieval on top of it removes you from the answer layer without gaining you anything on the training side.

What blocking actually costs

Hangcheng Zhao of Rutgers Business School and Ron Berman of Wharton examined publishers that blocked AI crawlers and found measurable traffic declines, including in human-only browsing rather than bot traffic alone.^[4] Treat the effect size with care: the paper has been revised repeatedly since the December 2025 preprint and the reported magnitude has moved between versions, so cite the direction rather than a specific percentage.

On the other side of the ledger, training crawlers consume far more than they return. Published crawl-to-referral ratios vary widely depending on the provider, the quarter and who is doing the measuring, but every version of the figure shows training crawlers taking many thousands of page fetches for each visitor sent back. If your bandwidth bill is the problem, that is a real reason to act, and it is a reason that has nothing to do with citations.

Worth being clear about what blocking cannot do. It does not remove content already absorbed into a trained model. It is a request rather than an enforcement mechanism, formalized in RFC 9309 and dependent on voluntary compliance. And it does not reliably stop user-triggered fetches, which several providers treat as a person retrieving a page rather than a crawler harvesting one.

The block you did not mean to set

Before adjusting anything, find out what your site is currently doing, because the file and the server frequently disagree. A July 2026 sample of the top 1,000 websites found 17.6% permitting GPTBot in robots.txt while returning 403 Forbidden when GPTBot actually requested a page, with the firewall or CDN silently overriding the published policy.^[5] A separate July 2026 measurement across 10,894 domains found the opposite mismatch too, with a large share of declared GPTBot bans not enforced at the server at all.^[8]

Nobody in either group knows which state they are in, because robots.txt is the thing people read and the response code is the thing that matters.

The fastest way to check is our free HTTP Header Checker, which fetches any URL and shows you the status code, the full redirect chain and the x-robots-tag your server actually returns, including the server-level noindex that never appears in your HTML. No account, no credits. If you would rather do it from a terminal, and you should if you want to compare several crawlers side by side:

# Verify what your server actually returns to each AI crawler.
# Run from any machine. Compare the status codes, not the robots.txt file.

# 1. What a normal browser gets (your control)
curl -sI -A "Mozilla/5.0" https://example.com/ | head -n 1

# 2. What OpenAI's training crawler gets
curl -sI -A "GPTBot/1.2" https://example.com/ | head -n 1

# 3. What OpenAI's search crawler gets. This one decides
#    whether you can appear in ChatGPT search answers.
curl -sI -A "OAI-SearchBot/1.0" https://example.com/ | head -n 1

# 4. Anthropic's search crawler
curl -sI -A "Claude-SearchBot" https://example.com/ | head -n 1

# 5. Perplexity's indexer
curl -sI -A "PerplexityBot/1.0" https://example.com/ | head -n 1

# Expected: 200 for every crawler you intend to allow.
# A 403 or 429 on line 3 means you are invisible in ChatGPT search
# regardless of what your robots.txt says.
# A 200 on a crawler you meant to block means your directive is
# declarative only, and you need a WAF rule to enforce it.

Run it against your own domain before you touch a single directive. Roughly one site in six is in a state its owner did not choose.

Two of our tools cover this without the terminal. The HTTP Header Checker shows what your server returns for a given URL, which is where the 403s and the hidden x-robots-tag headers turn up. The AI Visibility Checker tests whether AI crawlers can reach your pages at all. Both are free with no signup.

Making the call

One scheduling note. Cloudflare's default crawler settings change on September 15, and free-plan customers are among those affected, which means some sites will have their answer to this question changed for them. If you are on Cloudflare, check your settings before then rather than after. We covered the mechanics of that change separately.

Whatever you choose, verify it at the server rather than in the file, and check your citation trend afterwards rather than assuming. Crawler access tells you whether an engine can read you. It does not tell you whether one is citing you, and the two questions need separate answers. Our AI visibility tracking runs your prompt sets against ChatGPT, Google AI Overviews, Gemini and Perplexity, so a crawler change either shows up in the line or it does not. Pay-per-use, no subscription.