Do AI crawlers respect robots.txt? What the data shows

Mostly, with two big exceptions. In controlled tests, the major training and search crawlers (GPTBot, ClaudeBot, PerplexityBot) honor robots.txt. User-triggered fetchers such as ChatGPT-User and Perplexity-User skip it by design, and ByteDance's Bytespider ignores it outright. robots.txt is a published request, not an enforcement mechanism, so compliance is always the crawler's choice.

What does robots.txt actually promise?

Nothing enforceable. RFC 9309, the 2022 standard behind robots.txt, defines rules that crawlers "are requested to honor" and states that these rules "are not a form of access authorization." A well-behaved crawler reads the file and complies. A badly behaved one reads the same file and crawls anyway, with no technical consequence.

The file dates back to 1994 but only became a formal standard in September 2022, as RFC 9309. Even the standard tells you not to lean on it: "The Robots Exclusion Protocol is not a substitute for valid content security measures."

How often the request gets honored is no longer a matter of opinion. A study presented at the ACM Internet Measurement Conference in October 2025 watched 130 self-declared bots against controlled robots.txt files for 40 days. Across all bots, complete disallow directives were followed 30.7% of the time, endpoint restrictions 31.0%, and crawl-delay 60.9% (IMC 2025 study). The stricter the rule, the lower the compliance.

Which AI crawlers comply, and which do not?

In the IMC 2025 measurement, GPTBot, ClaudeBot, and ChatGPT-User followed disallow directives 100% of the time, while ByteDance's Bytespider followed them 0% of the time. The household names behave under observation. Compliance breaks down in the long tail, and the same study found fewer than 40% of AI assistant and search crawlers re-checked robots.txt within a week.

Every quoted behavior in that table comes from the operator's own crawler page: OpenAI, Anthropic, Perplexity, and Google. ByteDance is the exception. We could find no public crawler documentation for Bytespider, which fits its measured behavior.

Why do user-triggered fetchers skip robots.txt?

Because vendors classify a user asking about your page as browsing, not crawling. OpenAI's bot documentation says that for ChatGPT-User actions "robots.txt rules may not apply." Perplexity's docs say Perplexity-User "generally ignores robots.txt rules." Anthropic documents the opposite position and states that its bots honor do-not-crawl signals in robots.txt.

The practical consequence: a full disallow keeps your pages out of the training sets and search indexes of compliant operators, but it does not stop your content from surfacing when someone pastes your URL into a chat and asks about it. Neither vendor hides this. It sits in their public docs as a design position, and whether user-initiated fetching should count as crawling at all is one of the unsettled questions of the AI-crawl era.

What happened between Cloudflare and Perplexity?

In August 2025 Cloudflare reported that Perplexity, when blocked, switched to undeclared crawlers posing as Chrome on macOS, at 3 to 6 million requests per day across tens of thousands of domains. Perplexity denied it and attributed the traffic to a third-party service. The dispute was never independently settled.

Cloudflare's test was simple. It stood up fresh, unindexed domains with a robots.txt disallowing all crawlers, then asked Perplexity questions about those domains. The answers came back with detailed page content (Cloudflare, August 2025). OpenAI's ChatGPT-User, run through the same test, fetched the robots file and stopped. Perplexity denied the bots were its own, dismissed the post as a sales pitch, and said the flagged traffic came from a third-party service it uses (TechCrunch, August 2025).

Two things are true at once. The episode remains contested, and it still moved the market: Cloudflare pulled Perplexity from its verified-bots program, and network-level blocking gained its best sales argument yet.

Should you block AI crawlers?

Decide crawler by crawler, based on the job each one does. Blocking training bots such as GPTBot or ClaudeBot is a data-policy choice with no documented search cost. Blocking search and user fetchers costs visibility: OpenAI says sites that disallow OAI-SearchBot "will not be shown in ChatGPT search answers." A blanket block is a citation embargo.

Publishers lean toward blocking. As of July 2026, 631 of the 1,154 news sites tracked by the News Homepages project, 54.7%, block at least one of the OpenAI, Google AI, or Common Crawl bots (live tracker). Their math is defensible: Cloudflare measured how many pages AI companies crawl for each visitor they refer back, and for the first week of August 2025 the ratio was roughly 50,000 crawls per referral for Anthropic, 887 for OpenAI, and 118 for Perplexity (Cloudflare data). If your business is selling attention, that trade looks terrible.

If your business is being found and trusted, the math flips. An AI answer that cites you is a recommendation delivered at the moment of a buyer's question, which is the whole game of answer engine optimization, and the fetchers that produce those citations are exactly the ones a blanket block turns away. One cheap win either way: Google states that Google-Extended "does not impact a site's inclusion in Google Search," so opting out of Gemini training costs nothing in search. The allow-list that favors citations is in How to get cited by ChatGPT.

What actually stops a non-compliant crawler?

Infrastructure. CDN and WAF bot management identifies crawlers by traffic fingerprint and blocks them whatever user agent they claim. Newer preference standards do not change the enforcement picture: Cloudflare's own Content Signals Policy states that content signals "express preferences; they are not technical countermeasures against scraping."

If a crawler ignores your robots.txt, the fix lives at the network layer, where a request can be refused rather than politely asked to leave. Cloudflare's bot management is the widest-deployed example, and it identifies scrapers by behavior even "when operators lie about their user agent" (Cloudflare, July 2024). The practical stack for most sites: robots.txt for the compliant majority, bot management for the rest, and its September 2025 Content Signals Policy extension if you want machine-readable nuance between search, AI input, and AI training.

Where does that leave your own file? Somewhere measurable. robots.txt is the first thing an answer engine touches on your domain, and a misconfigured one either leaks content you meant to withhold or embargoes the citations you wanted. A free BeCited report checks whether the major AI crawlers can reach your pages and scores what they find, and the methodology page lists exactly what is measured live. If the fetcher that would have cited you is blocked, you find out from the report today instead of from six flat months of traffic.