Show HN AI Crawler Audit: 156 Launches, One Pasted robots.txt

Viewfy's show hn ai crawler audit probed 156 August 2026 Show HN launches. 23 (14.7%) forbid Anthropic's crawler while allowing OpenAI's search crawler. None do the reverse. Almost none wrote that file themselves.

What is the most surprising number in this audit?

The perfect one-sidedness. 23 sites (14.7%) disallow ClaudeBot in robots.txt while leaving OAI-SearchBot allowed, and 0 sites (0.0%) do the opposite. In a cohort of 156 independent founders shipping independent products, a coin-flip preference does not come out 23 to zero. Something is pasting the same decision into everyone's repo.

It is Cloudflare's managed robots.txt. Of the 24 sites in the cohort that disallow any AI crawler, 23 (95.8%) serve that managed file, identifiable by its Content-Signal preamble. The file declares search=yes in the same breath that it disallows ClaudeBot, GPTBot and Google-Extended. So the site is telling AI search engines it wants to be found, and telling three named crawlers to go away.

The cost is not abstract. atlasmotion.com launched motors for drones and robotics with 406 words of readable homepage HTML and one point on Hacker News. That page is unreachable to ChatGPT, Claude, Google AI Overviews and Perplexity at once. zipbox.ai shipped 2,141 words about Firecracker VMs for agents and has the same four-engine verdict. Nobody typed that file.

Who actually wrote the blocks in these robots.txt files?

Cloudflare did, on 33 of the 156 sites (21.2%). That is 50.8% of the 59 Cloudflare-fronted launches in the cohort, and Cloudflare fronts 37.8% of the cohort overall. Exactly one site in 156 runs a blanket disallow, which is the only block in the whole dataset that reads like a deliberate choice.

The written block is also weirdly uniform. ClaudeBot, GPTBot and Google-Extended are each disallowed on the same 24 sites (15.4%). OAI-SearchBot, ChatGPT-User, Claude-User, Claude-SearchBot and PerplexityBot are each disallowed on 1 (0.6%). A hand-written file would not produce that shape.

Half of it is theater. On 12 of the 23 managed-file sites, the origin still hands ClaudeBot a 200. The block exists only on paper, which is worse than either alternative: engines that respect robots.txt stay away from a page the server was happy to serve. One founder in r/SEO put it plainly: "Then I ran a curl on /robots.txt and saw this block that I definitely didn't write".

This is the class of problem a manual SEO audit misses, because the file looks fine in a browser and the site ranks fine in Google.

What do twelve of the measured sites look like line by line?

Read the table top to bottom. Rows where robots.txt names GPTBot, ClaudeBot and Google-Extended but the 403 column is empty are paper-only blocks. Rows where the 403 column is long and the robots column is empty are silent CDN blocks. Rows with both are invisible to all four engines.

reelang.com is the row to stare at. Three readable words in the un-executed HTML, a managed robots.txt disallowing three crawlers, and a server refusing seven user-agents including Google-Extended. A language-learning product with speech-analyzed native speaker video, and an AI assistant asked about it has nothing to read.

Why is the ChatGPT block silent while the Google block is written down?

Because the two layers block opposite engines. robots.txt files in this cohort forbid ClaudeBot, GPTBot and Google-Extended. The CDN and WAF layer refuses OAI-SearchBot, ChatGPT-User, Claude-User, Claude-SearchBot and PerplexityBot at the network layer, where nothing is declared.

15 sites (9.6%) returned a non-200 to at least one AI crawler. OAI-SearchBot and PerplexityBot were refused on 14 each (9.0%). Google-Extended was refused on only 2 (1.3%), and on 13 of those 15 sites (86.7%) Google-Extended got a 200 while every other AI crawler got a 403.

Stack the layers and you get lopsided honesty. The cohort is unreachable to Google AI Overviews on 25 sites (16.0%), with only 4.0% of those blocks silent. It is unreachable to ChatGPT on 15 sites (9.6%), with 93.3% of those blocks silent: nothing in those sites' robots.txt says ChatGPT is unwelcome. Perplexity matches ChatGPT at 15 sites (9.6%) and 93.3% silent. Claude is the worst off at 28 sites (17.9%), and there at least 14.3% is silent, the rest declared.

A commenter in r/SEO described the intent behind the split: "Cloudflare blocks the training bots, not the ones that handle search/appearance on the platform. For example, Cloudflare will block GPTBot and allow OAI-SearchBot". The measurement says the bot-fight rules underneath do not honor that split.

Does your host decide whether AI can read your launch?

More than your craft does. Cloudflare-fronted launches are invisible at 47.5% of 59 sites. Vercel-fronted launches are invisible at 2.6% of 39. That is a ratio of 18.5, and Cloudflare accounts for 82.4% of every invisible site while making up 37.8% of the cohort. nginx sat at 10.0% invisible across 20 sites.

Each stack fails differently. Cloudflare's failure is the block: 8.5% JavaScript-only, only 6.8% missing robots.txt. Vercel's failure is absence: 28.2% serve no robots.txt at all, nginx 35.0%. The 10 launches on shared app-platform subdomains were 20.0% invisible, 20.0% JavaScript-only shells, and 80.0% with no robots.txt.

Cohort-wide, 21.8% of the 156 are invisible to at least one engine and 71.8% are fully clean. 15.4% are blocked in robots.txt, 9.6% refused at the network layer, 5.8% ship an empty JavaScript shell, 19.2% serve no robots.txt, and 9.0% are thin HTML. Median readable words in raw HTML: 687. Sitemap declared: 67.3%. GPTBot is named in robots.txt on 26.9%, ClaudeBot on 26.3%, Google-Extended on 25.6%, OAI-SearchBot on 9.6%, PerplexityBot on 12.8%.

How did we measure this?

Cohort: Hacker News Algolia search_by_date with tags=show_hn, newest first, no upvote filter, deduped by host, with code hosts, app stores, socials and doc hosts dropped. 274 posts scanned down to 170 hosts, 166 reachable, four unreachable, 156 on their own domain and 10 on shared app-platform subdomains. Posts span 2026-08-05T02:36:20Z to 2026-08-07T01:14:28Z; collected 2026-08-07T01:21:00Z.

Probe: nine requests per homepage and no crawl beyond it. robots.txt plus the homepage fetched once as a desktop Chrome control and once as each of seven AI retrieval user-agents (OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended), run with python -m scripts.tier1_scan --file sites.txt. A second pass re-fetched every robots.txt verbatim and re-parsed it with the production parser to separate ClaudeBot from Claude-User and GPTBot from OAI-SearchBot, since the scan row only records engine-level verdicts. Spot checks were reproduced by hand with curl -A on atlasmotion.com and zipbox.ai (403 to OAI-SearchBot and ClaudeBot, 200 to Google-Extended) and on fosbury.ai/robots.txt.

Limits: homepage only, one request per user-agent, one two-day slice of Show HN, so a rate-limit or transient 403 can read as a permanent block, and we made no attempt to verify that a user-agent string belonged to the vendor it claims. This study counts ClaudeBot among retrieval crawlers and GPTBot as training-only; the 23 sites disallowing ClaudeBot still allow Claude-User and Claude-SearchBot, so their exposure to Claude is reduced, not zero. Reddit was read read-only across 334 public posts in r/SEO, r/startups, r/SaaS and r/Entrepreneur and 4 full threads; quotes are verbatim with permalinks. Raw artifacts (sites.txt, cohort.json, scan.jsonl, scan.summary.json, robots/.txt, robots.json, reddit/.json) sit in the workspace and stats.py recomputes every number from them.

What should you check on your own site this week?

In the order the data says you're likely to be hit, four checks, none longer than a coffee.

1. curl https://yoursite.com/robots.txt and look for a Content-Signal preamble. 21.2% of the cohort has one, and 50.8% of Cloudflare-fronted sites do. If it disallows ClaudeBot, GPTBot and Google-Extended while declaring search=yes, you did not write that and you can turn it off in the Cloudflare dashboard.
2. curl -A "OAI-SearchBot" -I https://yoursite.com and repeat for ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot and Google-Extended. 9.6% of the cohort returns a non-200 to at least one of them, and 93.3% of the ChatGPT blocks we found appear nowhere in robots.txt. Bot-fight modes and WAF rules do this quietly.
3. curl -s https://yoursite.com | wc -w on the un-executed HTML. 5.8% of launches ship a shell that is empty until JavaScript runs. Median in the cohort was 687 readable words. scalequest.io shipped five, reelang.com three.
4. Confirm robots.txt and sitemap.xml exist at all. 19.2% of the cohort serves no robots.txt, and only 67.3% declare a sitemap. On Vercel that miss rate was 28.2%, on nginx 35.0%.

The same single-site check is free at viewfy.ai/check. If you'd rather have the fix arrive as a diff than a to-do list, that is what the GitHub marketing integration does: robots.txt, sitemap and prerender changes opened as a pull request on your repo. Recommendations create work. That's the whole argument for marketing automation that fits git workflows instead of another dashboard, and it's the cheapest of the user acquisition tools an early team can run on launch day.

Correction, added after publication

A founder in the cohort pushed back on the network layer numbers, and he was right. Our probes sent crawler user agents from an ordinary IP, so a 403 in that setup measures how a CDN treats an impersonator, not the real crawler. Verified bots calling from their published IP ranges typically pass. The robots.txt findings in this post are unaffected, that file is public text anyone can read. But treat every "unreachable" and WAF number above as "unmeasurable from the outside", not "blocked". We have changed the methodology so outbound claims only use the robots.txt layer.