Who Blocks AI tracker › AI companies
AI Companies AI Visibility Report 2026

ChatGPT’s website blocks Perplexity. Claude’s blocks ChatGPT. Perplexity’s blocks both.

We tested 115 websites of 99 AI companies as 33 different AI crawlers. Seven of the 21 companies that run a crawler of their own - OpenAI, Anthropic, Microsoft, Meta, Amazon, Perplexity and Mistral AI - block a rival’s crawler by name. The other fourteen operators, Google and Apple among them, block nobody, and 78 of the 99 corporate sites admit every crawler tested.

For journalists: quote and reuse with credit under CC BY 4.0 Full CSV Who-blocks-whom PNG Crawler chart PNG Press contact

Key stats to quote

Copy any line with the source attached - free to use with credit.

ChatGPT’s website blocks Perplexity. Claude’s blocks ChatGPT. Perplexity’s blocks both.Three companies, three rules that name each other’s crawlers - each reproducible with one curl command.
7 of the 21 companies that run an AI crawler block a rival’s crawler by name: OpenAI, Anthropic, Microsoft, Meta, Amazon, Perplexity and Mistral AI.The companies whose products depend on an open web are the ones closing it to each other.
claude.ai blocks OpenAI’s crawlers in robots.txt. chatgpt.com blocks Perplexity’s in robots.txt. perplexity.ai blocks OpenAI’s and Anthropic’s at the edge.A circular firing squad - and every rule is reproducible with a single curl command.
PerplexityBot is shut out of 15 of the 115 AI-company sites tested - more than any other AI search crawler.The industry’s most blocked search crawler belongs to one of its loudest search players.
78 of the 99 AI companies’ corporate websites admit every one of the 33 AI crawlers tested.The blocking is concentrated at the top of the industry, not spread across it.
45 of the 99 AI companies publish a valid llms.txt. None of the eight largest - OpenAI, Anthropic, Google, Microsoft, Meta, Amazon, Apple and Perplexity - does.The machine-readability standard the ecosystem promotes has zero adoption among its biggest names.
11 of the 115 sites admit only crawlers on Cloudflare’s verified-bot list.A vendor’s list, not company policy, decides which AI crawlers can read the AI industry’s own sites.
The story in six findings

The AI industry is open to AI crawlers - except to each other’s.

Six numbers carry the report: who blocks a rival, whose crawler gets blocked most, how open the industry is, where the strictness lives, who adopts llms.txt, and who controls the gate.

7 / 21

crawler operators block a rival’s crawler by name

OpenAI, Anthropic, Microsoft, Meta, Amazon, Perplexity and Mistral AI each shut at least one rival out in robots.txt or at the edge. Fourteen operators, Google and Apple among them, block nobody.

Why it mattersThe crawler wars are between the household names, not the long tail.
15 / 115

sites shut PerplexityBot out - the most of any AI search crawler

Six block it by name, including chatgpt.com, meta.ai and amazon.com. Nine more refuse it because Cloudflare’s verified-bot list does not include it.

Why it mattersThe most blocked AI search crawler belongs to an AI search company.
78 / 99

corporate sites are open to every AI crawler tested

9 sit behind bot defences, 3 restrict training crawlers only, and 9 name an AI search or assistant crawler in a block rule.

Why it mattersThe blocking is concentrated at the top of the industry, not spread across it.
6 / 16

product domains block a rival’s crawler by name

chatgpt.com, claude.ai, ai.meta.com, meta.ai, chat.mistral.ai and grok.com - against 10 of the 99 corporate sites.

Why it mattersThe marketing site stays open; the app that holds the conversations does not.
45 / 99

publish a valid llms.txt

None of the eight largest AI companies does, and 9 more answer the URL with something that is not an llms.txt.

Why it mattersThe standard the ecosystem promotes has zero adoption among its biggest names.
11 / 115

sites admit only crawlers on Cloudflare’s verified-bot list

The gatekeeper for Google-GeminiNotebook, Perplexity-User, PerplexityBot and every other crawler not on it (inferred from the list).

Why it mattersA vendor’s list, not company policy, decides which AI crawlers get in.
1
Who blocks whom

Seven AI giants block a rival’s crawler by name. Fourteen operators block nobody.

The matrix below shows what each company’s websites do to its rivals’ crawlers, corporate site and product domains together (chatgpt.com with openai.com, claude.ai with anthropic.com). A cell shows the worst thing any of the company’s sites does to any of that rival’s crawlers. Count xAI, whose crawler is not in the registry this test sends, and it is 8 of the 22 companies in the matrix.

The eight that block Each row is one company’s sites; each column is one rival operator’s crawlers · November 2026

Company (sites tested)OpenAIAnthropicGoogleMetaAmazonPerplexityMistral AICommon CrawlSmaller operators
OpenAIopenai.com, chatgpt.comOwnRetired namesBlocks by nameRetired namesNot verifiedBlocks by nameNot verifiedBlocks by nameNot verified
Anthropicanthropic.com, claude.ai, claude.comBlocks by nameOwnBlocks by nameAllowsNot verifiedNot verifiedNot verifiedNot verifiedNot verified
Microsoftmicrosoft.com, copilot.microsoft.com, bing.comAllowsAllowsBlocks by nameAllowsAllowsAllowsAllowsAllowsBlocks by name
Metameta.com, ai.meta.com, meta.aiBlocks by nameBlocks by nameBlocks by nameOwnBlocks by nameBlocks by nameBlocks allBlocks allBlocks all
Amazonamazon.com, aws.amazon.comBlocks by nameBlocks by nameBlocks by nameBlocks by nameOwnBlocks by nameBlocks by nameBlocks by nameBlocks by name
Perplexityperplexity.aiBlocks by nameBlocks by nameAllowsBlocks by nameBot defenceOwnBot defenceAllowsBlocks by name
Mistral AImistral.ai, chat.mistral.aiBlocks by nameAllowsBlocks by nameAllowsNot verifiedNot verifiedOwnNot verifiedNot verified
xAIno crawler in registryx.ai, grok.comBlocks by nameBlocks by nameNot verifiedAllowsBlocks by nameNot verifiedAllowsBlocks by nameNot verified
Blocks by nameBlocks by name robots.txt or an edge rule naming the crawlerBlocks allBlocks all Disallow: / for every user agentNot verifiedNot verified refused by Cloudflare bot protection, not on its verified-bot list (inferred)Bot defenceBot defence challenge or bot management, no rule naming the crawlerRetired namesRetired names rules for old crawler names; the live crawler gets in

Hover a cell for the crawlers behind it.

The other 14 operators allow every rival crawler tested: GoogleGoogle DeepMindAppleYou.comDuckDuckGoCommon CrawlAi2 (Allen Institute)DiffbotKagiTimpiHive (ImageSift)Webz.ioBraveCohere
The rules, line by lineEvery block rule quoted, with the one-command evidence - for the technically minded

Each block is a rule that names the crawler - not a blanket bot defence. Open any rule for the exact commands; the company table carries the per-crawler verdicts for all 115 sites.

OpenAI → Google, Perplexity and Common Crawlchatgpt.com’s robots.txt disallows Google-Extended, PerplexityBot and CCBot. Its rules for the retired names anthropic-ai, Claude-Web, Omgilibot, Omgili and FacebookBot match nothing.
Show evidence (2 commands) PerplexityBot disallowed (line 40: Disallow: /):
curl -s 'https://chatgpt.com/robots.txt' | sed -n '39,40p'
CCBot disallowed (line 4: Disallow: /):
curl -s 'https://chatgpt.com/robots.txt' | sed -n '3,4p'
Anthropic → OpenAI and Googleclaude.ai’s robots.txt disallows OAI-SearchBot, ChatGPT-User, GPTBot and Google-Extended.
Show evidence (1 command) OAI-SearchBot, ChatGPT-User and GPTBot disallowed (line 10: Disallow: /):
curl -s 'https://claude.ai/robots.txt' | sed -n '6,10p'
Microsoft → Google and Webz.ioAn edge rule on microsoft.com refuses GoogleOther and webzio - every one of those names gets the edge’s block response.
Show evidence (2 commands) webzio served a different page instead of the site (HTTP 200) - the command prints its title, expect “Your request has been blocked. This could be due to several reasons.” (a browser gets “Microsoft: IA, nube, productividad, informática, juegos y aplicaciones”): Akamai also judges the client, so plain curl can get this page for every user agent from some networks; the clean reproduction is a browser - Chrome DevTools → Network conditions → User agent User agent → paste the agent above and load the page.rarr; paste the user agent from the command and load the page.
curl -s -A 'webzio' 'https://www.microsoft.com/es-es' | grep -o '<title>[^<]*'
GoogleOther served a different page instead of the site (HTTP 200) - the command prints its title, expect “Your request has been blocked. This could be due to several reasons.” (a browser gets “Microsoft: IA, nube, productividad, informática, juegos y aplicaciones”): Akamai also judges the client, so plain curl can get this page for every user agent from some networks; the clean reproduction is a browser - Chrome DevTools → Network conditions → User agent User agent → paste the agent above and load the page.rarr; paste the user agent from the command and load the page.
curl -s -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GoogleOther) Chrome/W.X.Y.Z Safari/537.36' 'https://www.microsoft.com/es-es' | grep -o '<title>[^<]*'
Meta → OpenAI, Anthropic, Google, Amazon, Apple and Perplexitymeta.ai’s robots.txt disallows GPTBot, ClaudeBot, Google-Extended, Amazonbot, Applebot-Extended and PerplexityBot - then Disallow: / for everyone else, Meta’s own crawlers included. An edge rule on meta.com and ai.meta.com refuses Google-GeminiNotebook.
Show evidence (3 commands) PerplexityBot disallowed (line 22: Disallow: /):
curl -s 'https://meta.ai/robots.txt' | sed -n '8,22p'
The catch-all group - Disallow: / for every other user agent (line 52: Disallow: /):
curl -s 'https://meta.ai/robots.txt' | sed -n '51,52p'
Google-GeminiNotebook refused by the edge rule - expect HTTP 400:
curl -sSI -A 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36 (compatible; Google-GeminiNotebook; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-gemininotebook)' 'https://www.meta.com/gb/'
Amazon → 14 operatorsamazon.com’s robots.txt disallows OAI-SearchBot, ChatGPT-User, GPTBot, Claude-SearchBot, Claude-User, ClaudeBot, Google-CloudVertexBot, GoogleOther and 14 more. Its rules for the retired names omgili and Claude-Web match nothing.
Show evidence (3 commands) OAI-SearchBot disallowed (line 277: Disallow: /):
curl -s 'https://amazon.com/robots.txt' | sed -n '276,277p'
Claude-SearchBot disallowed (line 253: Disallow: /):
curl -s 'https://amazon.com/robots.txt' | sed -n '252,253p'
PerplexityBot disallowed (line 151: Disallow: /):
curl -s 'https://amazon.com/robots.txt' | sed -n '150,151p'
Perplexity → OpenAI, Anthropic, Meta and You.comAn edge rule on perplexity.ai refuses GPTBot, ClaudeBot, meta-externalagent and YouBot - every one of those names gets the edge’s block response.
Show evidence (3 commands) YouBot refused by the edge rule - expect HTTP 403:
curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; YouBot/1.0; +https://docs.you.com/youbot; env:prod) Chrome/X.X.X.X Safari/537.36' 'https://www.perplexity.ai/'
GPTBot refused by the edge rule - expect HTTP 403:
curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot' 'https://www.perplexity.ai/'
ClaudeBot refused by the edge rule - expect HTTP 403:
curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; [email protected])' 'https://www.perplexity.ai/'
Mistral AI → OpenAI and Googlechat.mistral.ai’s robots.txt disallows GPTBot and Google-Extended.
Show evidence (1 command) GPTBot disallowed (line 2: Disallow: /):
curl -s 'https://chat.mistral.ai/robots.txt' | sed -n '1,2p'
xAI → OpenAI, Anthropic, Amazon and Common CrawlAn edge rule on grok.com refuses GPTBot, ClaudeBot, Amazonbot and CCBot - every one of those names gets the edge’s block response.
Show evidence (3 commands) GPTBot refused by the edge rule - expect HTTP 403:
curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot' 'https://grok.com/'
ClaudeBot refused by the edge rule - expect HTTP 403:
curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; [email protected])' 'https://grok.com/'
CCBot refused by the edge rule - expect HTTP 403:
curl -sSI -A 'CCBot/2.0 (https://commoncrawl.org/faq/)' 'https://grok.com/'
Matthew Woodward

“AI companies rely on an open web to power their products, but our research shows that seven are actively blocking competitors by name. You can’t champion an open web, whilst closing the door on your rivals.”

Matthew Woodward, founder, LinksThatRank

Why this is newsworthy

The companies whose products depend on crawling the open web are the only ones in the industry closing it to each other.

A cell shows the worst verdict across all of a company’s tested sites. “Not verified” is an inference from Cloudflare’s published verified-bot list, not an observed rule - see the methodology.

Download the who-blocks-whom PNG
Operators blocking a rival by name7 of 21
Operators blocking nobody14 of 21
Matrix companies blocking, with xAI8 of 22
2
The Perplexity problem

Perplexity runs the most blocked AI search crawler - and blocks OpenAI, Anthropic and Meta itself.

PerplexityBot is refused on 15 of the 115 sites tested: six by a rule that names it, nine more because those sites’ Cloudflare bot protection admits only crawlers on Cloudflare’s verified list, which Perplexity’s is not (an inference from that list - see the method). In the other direction, perplexity.ai refuses four rivals’ crawlers by name with HTTP 403 from both vantage points.

The Perplexity problem: the AI search crawlers shut out most, and what perplexity.ai does to its rivals' crawlersOpen the chart (PNG)

The 15 sites that shut PerplexityBot out

Six name it in a rule; nine admit only Cloudflare-verified crawlers.

Blocked by name
chatgpt.commeta.aiamazon.combaidu.comwriter.compoe.com
Refused as not Cloudflare-verified (inferred)
openai.comclaude.aichat.mistral.aix.aigenspark.aigamma.appmidjourney.comlovable.devcharacter.ai
Context: only the training crawlers (CCBot and the image and data-set bots) are shut out of more sites - and those are opt-outs from model training, not from search (see the crawlers shut out most).

Why this is newsworthy

Both directions at once: the industry shuts Perplexity’s crawler out more than any other search bot, and Perplexity shuts out OpenAI, Anthropic, Meta and You.com by name.

Download the Perplexity chart PNG
Sites refusing PerplexityBot15 of 115
By a rule naming it6
Rivals perplexity.ai blocks by name4
3
Open vs locked

Most of the AI industry is open. The apps are where the blocking happens.

78 of the 99 corporate websites answered every one of the 33 crawler identities with the page. The strictness is in the product domains: 6 of the 16 apps tested block a rival by name - chatgpt.com, claude.ai, ai.meta.com, meta.ai, chat.mistral.ai and grok.com - against 10 of the 99 corporate sites. The pattern is consistent: the marketing site stays open, the app that holds the conversations does not.

How the 99 AI companies' corporate websites answer AI crawlersOpen the chart (PNG)

How the 99 corporate sites answered AI crawlers

78
Fully openServed normal pages to every crawler identity tested.
7
Verified bots onlyCloudflare bot protection admits only crawlers on its verified list. more
5
robots.txt blocks AI searchrobots.txt disallows at least one AI search crawler.
3
Training onlyRestrict the training crawlers only; AI search and assistants get in.
2
Blocks AI assistantsAn edge rule names an on-demand assistant fetcher and refuses it.
2
Edge blocks AI searchAn edge rule names an AI search crawler and refuses it.
1
Bot managementRefuses bot traffic it does not recognise - no rule names an AI crawler.
1
ChallengedA JavaScript challenge in front of some or every non-browser client.

Why this is newsworthy

The AI industry keeps its shop windows open and locks the rooms where the conversations happen.

“Corporate site” is the domain classified and counted; product domains are reported alongside it and never change a company’s class.

Download the access chart PNG
Corporate sites open to every crawler78 of 99
Product domains blocking a rival6 of 16
Corporate sites blocking a rival by name10 of 99
4
Infrastructure

Cloudflare’s verified-bot list decides which AI crawlers get into 11 of the sites.

Seven corporate sites and four product domains run Cloudflare bot protection that challenges every crawler it cannot verify. Crawlers on Cloudflare’s published list get through - OAI-SearchBot, ClaudeBot, Amazonbot, DuckAssistBot and the rest. The ones not on it, Google-GeminiNotebook, Perplexity-User, PerplexityBot among them, get the same challenge our test did.

The crawlers shut out most

Sites turning each crawler away across all 115, by name or as not Cloudflare-verified (inferred) · top 12.

CCBot Common Crawl18
ImagesiftBot Imagesift15
PerplexityBot Perplexity15
AI2Bot Allen Institute for AI (Ai2)14
Timpibot Timpi14
Google-GeminiNotebook Google14
Diffbot Diffbot13
MistralAI-Training Mistral AI12
webzio Webz.io12
GPTBot OpenAI11
Amzn-User Amazon10
Perplexity-User Perplexity10
Blocked by nameRefused as not Cloudflare-verified (inferred)
Read the floor, not the ceiling: training crawlers are sent last in each check and time out first on slow sites, so their counts are a floor.

Who fronts the 99 corporate sites

43/99
Cloudflare
22/99
Vercel
11/99
No CDN / WAF detected
6/99
AWS CloudFront / WAF
5/99
Google Cloud
5/99
Origin (nginx)
2/99
Akamai
2/99
Origin (Apache)
1/99
Azure Front Door
1/99
Squarespace / Fastly each

Why this is newsworthy

Whether a new AI crawler can read the AI industry’s own websites increasingly comes down to one vendor’s verified-bot list.

“Not verified” verdicts are an inference from Cloudflare’s published list; the observed fact is that the site challenged our copy of the crawler and a spoofed Googlebot alike.

Download the crawlers-shut-out PNG
Sites admitting verified bots only11 of 115
Cloudflare in front of corporate sites43 of 99
Vercel in front of corporate sites22 of 99
5
llms.txt

45 of 99 AI companies publish an llms.txt. None of the eight largest does.

The industry adopts the machine-readability standard it asks of everyone else - up to a point. Nine more companies answer the llms.txt URL with an HTML page or an error instead of the file, and not one of the 99 publishes a valid TDM reservation file (tdmrep.json).

45
Publish a valid llms.txt45% of the 99 corporate sites serve a file that parses as llms.txt.
9
Answer the URL with something elseAn HTML page or an error where the file should be.
0
Of the eight largest have oneOpenAI, Anthropic, Google, Microsoft, Meta, Amazon, Apple, Perplexity.
0
Publish a valid tdmrep.jsonNo company serves a valid TDM reservation file.

Among the adopters:

Google DeepMindMistral AIYou.comDiffbotHive (ImageSift)CohereDatabricksCrusoeSambaNovaTogether AIBasetenFireworks AI+ 33 more

Why this is newsworthy

The AI ecosystem’s own machine-readability standard has 45% adoption among AI companies - and zero among the eight biggest.

Valid llms.txt45 of 99
Of the eight largest0 of 8
Valid tdmrep.json0 of 99
6
The robots.txt layer

Training opt-outs are small, and the robots.txt files still block crawlers that no longer exist.

Eleven of the 99 corporate sites restrict the crawlers that collect training data - seven in robots.txt, four with an edge rule only (Microsoft, Perplexity, Tencent and Writer) - and training policies never count against a company in this report. Meanwhile five sites still carry rules for retired crawler names that match nothing, and six allow a crawler in robots.txt that their edge then refuses by name.

11
Restrict training crawlersGPTBot, ClaudeBot, CCBot, meta-externalagent, Amazonbot and the rest - 7 in robots.txt, 4 at the edge only.
3
Block all five big training botsThe only sites restricting every major training crawler at once.
5
Carry retired crawler nameschatgpt.com, amazon.com, reflection.ai, midjourney.com and 1 more disallow names like anthropic-ai, Claude-Web, Omgilibot, Omgili - rules that match nothing today. more
6
robots.txt says allow, edge says noThe file allows a crawler the edge refuses by name - invisible to a robots-only audit.
6
Name any AI crawler in robots.txtOnly six of the 99 corporate sites mention one at all.

Why this is newsworthy

A robots.txt-only audit of the AI industry reads rules that match nothing - and misses the blocks that live at the edge.

Sites with retired crawler names5
robots.txt allow vs edge refusal6
Corporate robots.txt naming any AI crawler6 of 99
Group by group

Every group is around three-quarters open - the blocking sits with the crawler operators themselves.

The 99 companies in three groups: the 21 whose crawlers this test sends, the Forbes AI 50 of 2026, and other notable labs and AI-first companies.

How each group of AI companies answers AI crawlersOpen the chart (PNG)

AI crawler operators 21 companies

76% fully open - 16 open, 1 verified crawlers only, 1 training only, 1 blocks AI assistants, 1 blocks AI search, 1 robots.txt blocks AI search.

The twist: the most open group contains every one of the industry’s by-name blockers.

Forbes AI 50 (2026) 47 companies

77% fully open - 36 open, 1 bot management, 5 verified crawlers only, 2 training only, 1 challenged, 2 robots.txt blocks AI search.

Pattern: the challenges here are Vercel and Cloudflare defaults, not AI policies.

Other notable AI companies 31 companies

84% fully open - 26 open, 1 verified crawlers only, 1 blocks AI assistants, 1 blocks AI search, 2 robots.txt blocks AI search.

Every company

All 99 AI companies, ranked by what they block.

All 115 tested sites: the answer to AI crawlers, what each blocks by name, the blocking layer, the training policy and llms.txt, with the per-crawler verdicts behind every row. Product domains sit under their company’s corporate site.

Ordered by what each company blocks: the companies that name a rival’s crawler in a rule first, then the sites that admit only Cloudflare-verified crawlers, the training-only opt-outs, the bot defences, and the open ones. Click a column to sort, use the filter to narrow the list, and open a row for the crawler-by-crawler verdicts. 16 product domains (the apps: chatgpt.com, claude.ai, meta.ai …) are marked product under their company’s corporate site (↳). Each row has its own link (the # beside the name). Download the CSV.

#CompanyGroupSiteAnswer to AI crawlersBlocks by nameCDN / WAFTrainingllms.txt
1 #US runs a crawler Crawler operators perplexity.ai Blocks AI searchYouBot You.com, OpenAI, Anthropic, Meta Cloudflare MixedGPTBot, ClaudeBot, meta-externalagent —
2 #US Other AI companies writer.com Blocks AI searchPerplexityBot, YouBot Perplexity, You.com, OpenAI, Anthropic +2 Cloudflare MixedGPTBot, ClaudeBot, CCBot, Amazonbot Yes
3 #US runs a crawler Crawler operators amazon.com robots.txt blocks searchOAI-SearchBot, Claude-SearchBot, PerplexityBot, meta-webindexer, DuckAssistBot, YouBot +4 more OpenAI, Anthropic, Perplexity, Meta +10 Akamai MixedGPTBot, ClaudeBot, CCBot, meta-externalagent… —
↳ #US product Crawler operators aws.amazon.com Open — AWS CloudFront / WAF Open —
4 #US runs a crawler Crawler operators anthropic.com Open — Cloudflare Open —
↳ #US product Crawler operators claude.ai robots.txt blocks searchOAI-SearchBot, ChatGPT-User OpenAI, Google Cloudflare MixedGPTBot, Google-Extended Invalid
↳ #US product Crawler operators claude.com Open — Cloudflare Open —
5 #CN runs a crawler Other AI companies baidu.com robots.txt blocks searchOAI-SearchBot, Claude-SearchBot, PerplexityBot, meta-webindexer, Amzn-SearchBot, MistralAI-Index +7 more OpenAI, Anthropic, Perplexity, Meta +14Disallow: / for everyone No CDN / WAF detected RestrictedGPTBot, ClaudeBot, CCBot, meta-externalagent… —
6 #US runs a crawler Crawler operators meta.com Blocks assistantsGoogle-GeminiNotebook Google No CDN / WAF detected Open —
↳ #US product Crawler operators ai.meta.com Blocks assistantsGoogle-GeminiNotebook Google No CDN / WAF detected Open —
↳ #US product Crawler operators meta.ai robots.txt blocks searchOAI-SearchBot, Claude-SearchBot, PerplexityBot, meta-webindexer, Amzn-SearchBot, MistralAI-Index +7 more Perplexity, OpenAI, Anthropic, Amazon +14Disallow: / for everyone No CDN / WAF detected RestrictedGPTBot, ClaudeBot, CCBot, meta-externalagent… —
7 #US Forbes AI 50 midjourney.com robots.txt blocks searchYouBot You.com, OpenAI, Common Crawl, Meta +5 Cloudflare MixedGPTBot, CCBot, meta-externalagent, Amazonbot… —
8 #US Forbes AI 50 notion.com robots.txt blocks searchAmzn-SearchBot Amazon, Meta, Imagesift Cloudflare MixedImagesiftBot, meta-externalagent, Amazonbot Yes
9 #US runs a crawler Crawler operators openai.com Verified bots onlynot verified: PerplexityBot, Perplexity-User, Amzn-User… — Cloudflare Open —
↳ #US product Crawler operators chatgpt.com robots.txt blocks searchPerplexityBot Perplexity, Common Crawl, Google Cloudflare MixedCCBot, Google-Extended —
10 #US runs a crawler Other AI companies poe.com robots.txt blocks searchOAI-SearchBot, Claude-SearchBot, PerplexityBot, meta-webindexer, Amzn-SearchBot, MistralAI-Index +7 more OpenAI, Anthropic, Perplexity, Meta +14Disallow: / for everyone Cloudflare MixedClaudeBot, CCBot, meta-externalagent, Amazonbot… Invalid
11 #CN Other AI companies tencent.com Blocks assistantsGoogle-Agent, Google-GeminiNotebook Google, OpenAI, Anthropic, Common Crawl +9 Origin (nginx) MixedGPTBot, ClaudeBot, CCBot, meta-externalagent… —
12 #US runs a crawler Crawler operators microsoft.com Training only Webz.io, Google Azure Front Door Mixedwebzio, GoogleOther —
↳ #US product Crawler operators bing.com Open — No CDN / WAF detected Open —
↳ #US product Crawler operators copilot.microsoft.com Open — Azure Front Door Open Invalid
13 #FR runs a crawler Crawler operators mistral.ai Open — Cloudflare Open Yes
↳ #FR product Crawler operators chat.mistral.ai Verified bots onlynot verified: PerplexityBot, Perplexity-User, Amzn-User… OpenAI, Google Cloudflare MixedGPTBot, Google-Extended Invalid
14 #US Forbes AI 50 reflection.ai Training only OpenAI, Anthropic, Common Crawl, Meta +5 Cloudflare MixedGPTBot, ClaudeBot, CCBot, meta-externalagent… —
15 #US Forbes AI 50 together.ai Training only Google Cloudflare MixedGoogle-Extended Yes
16 #US Forbes AI 50 x.ai Verified bots onlynot verified: PerplexityBot, Perplexity-User, Google-GeminiNotebook… — Cloudflare Open —
↳ #US product Forbes AI 50 grok.com Training only OpenAI, Anthropic, Common Crawl, Amazon Cloudflare MixedGPTBot, ClaudeBot, CCBot, Amazonbot —
17 #US Other AI companies character.ai Verified bots onlynot verified: PerplexityBot, Perplexity-User, Amzn-User… — Cloudflare Open —
18 #US Forbes AI 50 databricks.com Verified bots onlynot verified: Google-GeminiNotebook — Cloudflare Open Yes
19 #US Forbes AI 50 gamma.app Verified bots onlynot verified: PerplexityBot, Perplexity-User, Amzn-User… — Cloudflare Open Yes
20 #US Forbes AI 50 genspark.ai Verified bots onlynot verified: PerplexityBot, Perplexity-User, Google-GeminiNotebook… — Cloudflare Open Yes
21 #SE Forbes AI 50 lovable.dev Verified bots onlynot verified: PerplexityBot, Perplexity-User, Amzn-User… — Cloudflare Open Yes
22 #CN Other AI companies deepseek.com Open — AWS CloudFront / WAF Open —
↳ #CN product Other AI companies chat.deepseek.com Verification WAF — AWS CloudFront / WAF Open —
23 #US Forbes AI 50 openevidence.com Bot management — Vercel Open —
24 #US Forbes AI 50 replit.com Challenged — Cloudflare Open Yes
25 #IL Other AI companies ai21.com Open — Cloudflare Open Yes
26 #US Forbes AI 50 abridge.com Open — Cloudflare Open —
27 #US runs a crawler Crawler operators allenai.org Open — Google Cloud Open —
28 #DE Other AI companies aleph-alpha.com Open — Origin (nginx) Open —
29 #CN Other AI companies qwen.ai Open — No CDN / WAF detected Open Invalid
30 #CN Other AI companies alibabacloud.com Open — No CDN / WAF detected Open Invalid
31 #US runs a crawler Crawler operators apple.com Open — No CDN / WAF detected Open —
32 #US Forbes AI 50 appliedintuition.com Open — Vercel Open Yes
33 #US Forbes AI 50 baseten.co Open — Vercel Open Yes
34 #DE Forbes AI 50 bfl.ai Open — Vercel Open Yes
35 #US runs a crawler Crawler operators brave.com Open — AWS CloudFront / WAF Open —
↳ #US product Crawler operators search.brave.com Open — AWS CloudFront / WAF Open —
36 #CN runs a crawler Crawler operators bytedance.com Open — No CDN / WAF detected Open —
37 #US Other AI companies cerebras.ai Open — Vercel Open —
38 #US Forbes AI 50 chaidiscovery.com Open — No CDN / WAF detected Open —
39 #US Forbes AI 50 clay.com Open — Cloudflare Open —
40 #US Forbes AI 50 cognition.com Open — Vercel Open Yes
↳ #US product Forbes AI 50 devin.ai Open — Vercel Open Yes
41 #CA runs a crawler Forbes AI 50 cohere.com Open — Vercel Open Yes
42 #US runs a crawler Crawler operators commoncrawl.org Open — Cloudflare Open —
43 #US Forbes AI 50 crusoe.ai Open — Cloudflare Open Yes
44 #US Forbes AI 50 cursor.com Open — Vercel Open Yes
45 #US Forbes AI 50 cyera.com Open — Cloudflare Open Yes
46 #US Forbes AI 50 decagon.ai Open — Cloudflare Open Yes
47 #US runs a crawler Crawler operators diffbot.com Open — Origin (nginx) Open Yes
48 #US runs a crawler Crawler operators duckduckgo.com Open — Origin (nginx) Open Invalid
49 #US Forbes AI 50 elevenlabs.io Open — Google Cloud Open Yes
50 #US Forbes AI 50 eliseai.com Open — Cloudflare Open Invalid
51 #US Forbes AI 50 fal.ai Open — Cloudflare Open Yes
52 #US Forbes AI 50 fireworks.ai Open — Vercel Open Yes
53 #US Forbes AI 50 glean.com Open — Cloudflare Open —
54 #US runs a crawler Crawler operators google.com Open — Google Cloud Open —
↳ #US product Crawler operators gemini.google.com Open — No CDN / WAF detected Open —
55 #US runs a crawler Crawler operators deepmind.google Open — Google Cloud Open Yes
56 #US Other AI companies grammarly.com Open — AWS CloudFront / WAF Open —
57 #US Other AI companies groq.com Open — Vercel Open —
58 #US Forbes AI 50 harvey.ai Open — Vercel Open —
59 #US Forbes AI 50 heygen.com Open — Cloudflare Open Yes
60 #US runs a crawler Crawler operators thehive.ai Open — AWS CloudFront / WAF Open Yes
61 #US Other AI companies huggingface.co Open — AWS CloudFront / WAF Open —
62 #US Other AI companies ideogram.ai Open — Cloudflare Open Yes
63 #US Other AI companies inflection.ai Open — No CDN / WAF detected Open —
64 #US Other AI companies jasper.ai Open — Cloudflare Open Yes
65 #US runs a crawler Crawler operators kagi.com Open — Google Cloud Open —
66 #US Forbes AI 50 krea.ai Open — Cloudflare Open Yes
67 #SE Forbes AI 50 legora.com Open — No CDN / WAF detected Open Yes
68 #US Other AI companies liquid.ai Open — Vercel Open —
69 #US Forbes AI 50 listenlabs.ai Open — Vercel Open Yes
70 #US Other AI companies lumalabs.ai Open — Vercel Open Yes
71 #SG Other AI companies manus.im Open — AWS CloudFront / WAF Open —
72 #US Forbes AI 50 mercor.com Open — Cloudflare Open Yes
73 #CN Other AI companies minimax.io Open — Akamai Open Invalid
74 #CN Other AI companies moonshot.cn Open — Origin (nginx) Open Invalid
↳ #CN product Other AI companies kimi.com Open — Cloudflare Open Invalid
75 #US Other AI companies nvidia.com Open — Origin (Apache) Open Yes
76 #US Other AI companies otter.ai Open — Cloudflare Open Yes
77 #US Forbes AI 50 pi.website Open — Vercel Open —
78 #US Other AI companies pika.art Open — Vercel Open Yes
79 #US Other AI companies replika.com Open — No CDN / WAF detected Open Yes
80 #US Forbes AI 50 rogo.com Open — Vercel Open Yes
81 #US Forbes AI 50 runway.com Open — Vercel Open Yes
82 #US Forbes AI 50 ssi.inc Open — Vercel Open —
83 #JP Other AI companies sakana.ai Open — Fastly Open —
84 #US Forbes AI 50 sambanova.ai Open — Cloudflare Open Yes
85 #US Other AI companies scale.com Open — Vercel Open —
86 #US Forbes AI 50 sierra.ai Open — Cloudflare Open —
87 #US Forbes AI 50 skild.ai Open — Cloudflare Open —
88 #US Forbes AI 50 speak.com Open — Cloudflare Open Yes
89 #GB Other AI companies stability.ai Open — Squarespace Open —
90 #US Forbes AI 50 suno.com Open — Cloudflare Open Yes
91 #US Forbes AI 50 surgehq.ai Open — Cloudflare Open —
92 #GB Forbes AI 50 synthesia.io Open — Cloudflare Open Yes
93 #US Forbes AI 50 thinkingmachines.ai Open — Cloudflare Open —
94 #US runs a crawler Crawler operators timpi.com Open — Origin (Apache) Open Invalid
95 #US Other AI companies udio.com Open — Vercel Open —
96 #IL runs a crawler Crawler operators webz.io Open — Cloudflare Open Yes
97 #US Forbes AI 50 worldlabs.ai Open — Vercel Open —
98 #US runs a crawler Crawler operators you.com Open — Cloudflare Open Yes
99 #CN Other AI companies zhipuai.cn Open — No CDN / WAF detected Open Invalid
↳ #CN product Other AI companies chat.z.ai Open — No CDN / WAF detected Open Invalid

Showing the first 3 companies. Filtering, sorting or a row link opens the whole table.

Would your site pass the same test?

Paste your domain into the free AI Visibility Checker: it runs the same 33-crawler test, names the layer that blocks each crawler and gives you the fix.

Run the free check

Press resources

Everything on this page may be quoted and reused with credit under CC BY 4.0. The dataset, charts and per-company breakdowns are free to download, and press requests get a same-day answer.

Full CSV115 sites · one row each · every verdict plus the robots.txt and llms.txt findings · 12 borderline big-tech rows marked as not counted Download CSV
Publication chartsWho blocks whom · The Perplexity problem · How the sites answer AI crawlers · The crawlers shut out most · Group by group
Bespoke cutsCompany, group and crawler breakdowns, plus the raw request logs behind any verdict, on request
Interview / commentPress requests answered the same working day: [email protected]

Cite as: LinksThatRank, AI Companies AI Visibility Report, November 2026, https://linksthatrank.com/ai-visibility-checker/which-ai-companies-block-ai-crawlers/

Methodology

How we tested: 99 companies, 115 sites, 33 crawler identities, two vantage points.

Everything a reporter, editor or technical reader needs to check exactly how each classification was made. The full method is on the methodology page.

Who99 AI companies in three groups as of 19 October 2026: the 21 companies whose crawlers the test sends, the Forbes AI 50 of 2026, and other notable labs and AI-first companies.