crawler operators block a rival’s crawler by name
OpenAI, Anthropic, Microsoft, Meta, Amazon, Perplexity and Mistral AI each shut at least one rival out in robots.txt or at the edge. Fourteen operators, Google and Apple among them, block nobody.
We tested 115 websites of 99 AI companies as 33 different AI crawlers. Seven of the 21 companies that run a crawler of their own - OpenAI, Anthropic, Microsoft, Meta, Amazon, Perplexity and Mistral AI - block a rival’s crawler by name. The other fourteen operators, Google and Apple among them, block nobody, and 78 of the 99 corporate sites admit every crawler tested.
Copy any line with the source attached - free to use with credit.
Six numbers carry the report: who blocks a rival, whose crawler gets blocked most, how open the industry is, where the strictness lives, who adopts llms.txt, and who controls the gate.
OpenAI, Anthropic, Microsoft, Meta, Amazon, Perplexity and Mistral AI each shut at least one rival out in robots.txt or at the edge. Fourteen operators, Google and Apple among them, block nobody.
Six block it by name, including chatgpt.com, meta.ai and amazon.com. Nine more refuse it because Cloudflare’s verified-bot list does not include it.
9 sit behind bot defences, 3 restrict training crawlers only, and 9 name an AI search or assistant crawler in a block rule.
chatgpt.com, claude.ai, ai.meta.com, meta.ai, chat.mistral.ai and grok.com - against 10 of the 99 corporate sites.
None of the eight largest AI companies does, and 9 more answer the URL with something that is not an llms.txt.
The gatekeeper for Google-GeminiNotebook, Perplexity-User, PerplexityBot and every other crawler not on it (inferred from the list).
The matrix below shows what each company’s websites do to its rivals’ crawlers, corporate site and product domains together (chatgpt.com with openai.com, claude.ai with anthropic.com). A cell shows the worst thing any of the company’s sites does to any of that rival’s crawlers. Count xAI, whose crawler is not in the registry this test sends, and it is 8 of the 22 companies in the matrix.
| Company (sites tested) | OpenAI | Anthropic | Meta | Amazon | Perplexity | Mistral AI | Common Crawl | Smaller operators | |
|---|---|---|---|---|---|---|---|---|---|
| Own | Retired names | Blocks by name | Retired names | Not verified | Blocks by name | Not verified | Blocks by name | Not verified | |
| Blocks by name | Own | Blocks by name | Allows | Not verified | Not verified | Not verified | Not verified | Not verified | |
| Allows | Allows | Blocks by name | Allows | Allows | Allows | Allows | Allows | Blocks by name | |
| Blocks by name | Blocks by name | Blocks by name | Own | Blocks by name | Blocks by name | Blocks all | Blocks all | Blocks all | |
| Blocks by name | Blocks by name | Blocks by name | Blocks by name | Own | Blocks by name | Blocks by name | Blocks by name | Blocks by name | |
| Blocks by name | Blocks by name | Allows | Blocks by name | Bot defence | Own | Bot defence | Allows | Blocks by name | |
| Blocks by name | Allows | Blocks by name | Allows | Not verified | Not verified | Own | Not verified | Not verified | |
| Blocks by name | Blocks by name | Not verified | Allows | Blocks by name | Not verified | Allows | Blocks by name | Not verified |
Hover a cell for the crawlers behind it.
Open the chart (PNG)Each block is a rule that names the crawler - not a blanket bot defence. Open any rule for the exact commands; the company table carries the per-crawler verdicts for all 115 sites.
curl -s 'https://chatgpt.com/robots.txt' | sed -n '39,40p'curl -s 'https://chatgpt.com/robots.txt' | sed -n '3,4p'curl -s 'https://claude.ai/robots.txt' | sed -n '6,10p'curl -s -A 'webzio' 'https://www.microsoft.com/es-es' | grep -o '<title>[^<]*'curl -s -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GoogleOther) Chrome/W.X.Y.Z Safari/537.36' 'https://www.microsoft.com/es-es' | grep -o '<title>[^<]*'Disallow: / for everyone else, Meta’s own crawlers included. An edge rule on meta.com and ai.meta.com refuses Google-GeminiNotebook.
curl -s 'https://meta.ai/robots.txt' | sed -n '8,22p'curl -s 'https://meta.ai/robots.txt' | sed -n '51,52p'curl -sSI -A 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36 (compatible; Google-GeminiNotebook; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-gemininotebook)' 'https://www.meta.com/gb/'curl -s 'https://amazon.com/robots.txt' | sed -n '276,277p'curl -s 'https://amazon.com/robots.txt' | sed -n '252,253p'curl -s 'https://amazon.com/robots.txt' | sed -n '150,151p'curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; YouBot/1.0; +https://docs.you.com/youbot; env:prod) Chrome/X.X.X.X Safari/537.36' 'https://www.perplexity.ai/'curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot' 'https://www.perplexity.ai/'curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; [email protected])' 'https://www.perplexity.ai/'curl -s 'https://chat.mistral.ai/robots.txt' | sed -n '1,2p'curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot' 'https://grok.com/'curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; [email protected])' 'https://grok.com/'curl -sSI -A 'CCBot/2.0 (https://commoncrawl.org/faq/)' 'https://grok.com/'“AI companies rely on an open web to power their products, but our research shows that seven are actively blocking competitors by name. You can’t champion an open web, whilst closing the door on your rivals.”
Matthew Woodward, founder, LinksThatRankThe companies whose products depend on crawling the open web are the only ones in the industry closing it to each other.
A cell shows the worst verdict across all of a company’s tested sites. “Not verified” is an inference from Cloudflare’s published verified-bot list, not an observed rule - see the methodology.
Download the who-blocks-whom PNGPerplexityBot is refused on 15 of the 115 sites tested: six by a rule that names it, nine more because those sites’ Cloudflare bot protection admits only crawlers on Cloudflare’s verified list, which Perplexity’s is not (an inference from that list - see the method). In the other direction, perplexity.ai refuses four rivals’ crawlers by name with HTTP 403 from both vantage points.
Six name it in a rule; nine admit only Cloudflare-verified crawlers.
Blocked by nameBoth directions at once: the industry shuts Perplexity’s crawler out more than any other search bot, and Perplexity shuts out OpenAI, Anthropic, Meta and You.com by name.
Download the Perplexity chart PNG78 of the 99 corporate websites answered every one of the 33 crawler identities with the page. The strictness is in the product domains: 6 of the 16 apps tested block a rival by name - chatgpt.com, claude.ai, ai.meta.com, meta.ai, chat.mistral.ai and grok.com - against 10 of the 99 corporate sites. The pattern is consistent: the marketing site stays open, the app that holds the conversations does not.
The AI industry keeps its shop windows open and locks the rooms where the conversations happen.
“Corporate site” is the domain classified and counted; product domains are reported alongside it and never change a company’s class.
Download the access chart PNGSeven corporate sites and four product domains run Cloudflare bot protection that challenges every crawler it cannot verify. Crawlers on Cloudflare’s published list get through - OAI-SearchBot, ClaudeBot, Amazonbot, DuckAssistBot and the rest. The ones not on it, Google-GeminiNotebook, Perplexity-User, PerplexityBot among them, get the same challenge our test did.
Sites turning each crawler away across all 115, by name or as not Cloudflare-verified (inferred) · top 12.
Whether a new AI crawler can read the AI industry’s own websites increasingly comes down to one vendor’s verified-bot list.
“Not verified” verdicts are an inference from Cloudflare’s published list; the observed fact is that the site challenged our copy of the crawler and a spoofed Googlebot alike.
Download the crawlers-shut-out PNGThe industry adopts the machine-readability standard it asks of everyone else - up to a point. Nine more companies answer the llms.txt URL with an HTML page or an error instead of the file, and not one of the 99 publishes a valid TDM reservation file (tdmrep.json).
Among the adopters:
The AI ecosystem’s own machine-readability standard has 45% adoption among AI companies - and zero among the eight biggest.
Eleven of the 99 corporate sites restrict the crawlers that collect training data - seven in robots.txt, four with an edge rule only (Microsoft, Perplexity, Tencent and Writer) - and training policies never count against a company in this report. Meanwhile five sites still carry rules for retired crawler names that match nothing, and six allow a crawler in robots.txt that their edge then refuses by name.
A robots.txt-only audit of the AI industry reads rules that match nothing - and misses the blocks that live at the edge.
The 99 companies in three groups: the 21 whose crawlers this test sends, the Forbes AI 50 of 2026, and other notable labs and AI-first companies.
76% fully open - 16 open, 1 verified crawlers only, 1 training only, 1 blocks AI assistants, 1 blocks AI search, 1 robots.txt blocks AI search.
The twist: the most open group contains every one of the industry’s by-name blockers.
77% fully open - 36 open, 1 bot management, 5 verified crawlers only, 2 training only, 1 challenged, 2 robots.txt blocks AI search.
Pattern: the challenges here are Vercel and Cloudflare defaults, not AI policies.
84% fully open - 26 open, 1 verified crawlers only, 1 blocks AI assistants, 1 blocks AI search, 2 robots.txt blocks AI search.
All 115 tested sites: the answer to AI crawlers, what each blocks by name, the blocking layer, the training policy and llms.txt, with the per-crawler verdicts behind every row. Product domains sit under their company’s corporate site.
Ordered by what each company blocks: the companies that name a rival’s crawler in a rule first, then the sites that admit only Cloudflare-verified crawlers, the training-only opt-outs, the bot defences, and the open ones. Click a column to sort, use the filter to narrow the list, and open a row for the crawler-by-crawler verdicts. 16 product domains (the apps: chatgpt.com, claude.ai, meta.ai …) are marked product under their company’s corporate site (↳). Each row has its own link (the # beside the name). Download the CSV.
| # | Company | Group | Site | Answer to AI crawlers | Blocks by name | CDN / WAF | Training | llms.txt |
|---|---|---|---|---|---|---|---|---|
| 1 | Crawler operators | perplexity.ai | Blocks AI searchYouBot | You.com, OpenAI, Anthropic, Meta | Cloudflare | MixedGPTBot, ClaudeBot, meta-externalagent | — | |
|
Cloudflare blocks bot traffic it does not recognise. An unknown, self-identifying bot got the same HTTP 403 as 7 AI crawlers, so this is bot management, not a rule naming them. Tested from 2 vantage points (2 usable) · robots.txt readable · silent blocks: YouBot, GPTBot, ClaudeBot, meta-externalagent · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot ChallengedMistralAI-Index ChallengedDuckAssistBot AllowedYouBot Blocked (named)Googlebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User ChallengedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot Blocked (named)ClaudeBot Blocked (named)CCBot Allowedmeta-externalagent Blocked (named)Amazonbot AllowedMistralAI-Training ChallengedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot ChallengedTimpibot Allowedwebzio ChallengedImagesiftBot AllowedKagibot ChallengedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
Show evidence (4 commands: every verdict other than a plain allow)YouBot · Blocked by name · HTTP 403 · http 403 behind cloudflare:curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; YouBot/1.0; +https://docs.you.com/youbot; env:prod) Chrome/X.X.X.X Safari/537.36' 'https://www.perplexity.ai/'curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot' 'https://www.perplexity.ai/'curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; [email protected])' 'https://www.perplexity.ai/'curl -sSI -A 'meta-externalagent/1.1' 'https://www.perplexity.ai/' | ||||||||
| 2 | Other AI companies | writer.com | Blocks AI searchPerplexityBot, YouBot | Perplexity, You.com, OpenAI, Anthropic +2 | Cloudflare | MixedGPTBot, ClaudeBot, CCBot, Amazonbot | Yes | |
|
Cloudflare is blocking 6 AI crawlers at the edge. Its robots.txt allows them. Tested from 2 vantage points (2 usable) · robots.txt readable · silent blocks: PerplexityBot, YouBot, GPTBot, ClaudeBot, CCBot, Amazonbot · llms.txt valid · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Blocked (named)meta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot Blocked (named)Googlebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot Blocked (named)ClaudeBot Blocked (named)CCBot Blocked (named)meta-externalagent AllowedAmazonbot Blocked (named)MistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
Show evidence (6 commands: every verdict other than a plain allow)PerplexityBot · Blocked by name · HTTP 403 · body: "Your request was blocked." (text/plain — Cloudflare AI Crawl Control policy response):curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)' 'https://writer.com/'curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; YouBot/1.0; +https://docs.you.com/youbot; env:prod) Chrome/X.X.X.X Safari/537.36' 'https://writer.com/'curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot' 'https://writer.com/'curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; [email protected])' 'https://writer.com/'curl -sSI -A 'CCBot/2.0 (https://commoncrawl.org/faq/)' 'https://writer.com/'curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1) Chrome/W.X.Y.Z Safari/537.36' 'https://writer.com/' | ||||||||
| 3 | Crawler operators | amazon.com | robots.txt blocks searchOAI-SearchBot, Claude-SearchBot, PerplexityBot, meta-webindexer, DuckAssistBot, YouBot +4 more | OpenAI, Anthropic, Perplexity, Meta +10 | Akamai | MixedGPTBot, ClaudeBot, CCBot, meta-externalagent… | — | |
|
Robots.txt disallows 10 AI search crawlers and assistants (OAI-SearchBot, Claude-SearchBot, PerplexityBot, meta-webindexer and 6 more); behind it, Akamai also keeps 5 more from the page (5 challenged: Applebot, Perplexity-User, Amzn-User, Google-Agent and 1 more). Tested from 2 vantage points (1 usable) · robots.txt readable · AI tokens in robots.txt: GPTBot, CCBot, PerplexityBot, Google-Extended, ClaudeBot, meta-externalagent, Bytespider, omgili, AI2Bot, MistralAI-User, Diffbot, Google-NotebookLM, DuckAssistBot, Ai2Bot-Dolma, Google-CloudVertexBot, meta-externalfetcher, cohere-ai, YouBot, Claude-User, Claude-SearchBot, Perplexity-User, ChatGPT-User, OAI-SearchBot, Timpibot, AI2Bot-DeepResearchEval, Claude-Web, cohere-training-data-crawler, GoogleOther, ImagesiftBot, meta-webindexer, webzio-extended · retired names: omgili, Claude-Web · checked 6 November 2026 OAI-SearchBot robots.txtClaude-SearchBot robots.txtPerplexityBot robots.txtmeta-webindexer robots.txtAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot robots.txtYouBot robots.txtGooglebot Allowedbingbot AllowedApplebot ChallengedChatGPT-User robots.txtClaude-User robots.txtPerplexity-User Challengedmeta-externalfetcher robots.txtMistralAI-User robots.txtAmzn-User ChallengedGoogle-Agent ChallengedGoogle-GeminiNotebook ChallengedGPTBot robots.txtClaudeBot robots.txtCCBot robots.txtmeta-externalagent robots.txtAmazonbot ChallengedMistralAI-Training ChallengedAI2Bot robots.txtGoogle-CloudVertexBot robots.txtDiffbot robots.txtTimpibot robots.txtwebzio AllowedImagesiftBot robots.txtKagibot ChallengedGoogleOther robots.txtGoogle-Extended robots.txtApplebot-Extended Allowedwebzio-extended robots.txtCoherebot Allowed
Show evidence (8 commands: every verdict other than a plain allow)OAI-SearchBot · Disallowed in robots.txt · line 277: Disallow: / · robots.txt:curl -s 'https://amazon.com/robots.txt' | sed -n '276,277p'curl -s 'https://amazon.com/robots.txt' | sed -n '252,253p'curl -s 'https://amazon.com/robots.txt' | sed -n '150,151p'curl -s 'https://amazon.com/robots.txt' | sed -n '390,391p'curl -s 'https://amazon.com/robots.txt' | sed -n '204,205p'curl -s 'https://amazon.com/robots.txt' | sed -n '240,241p'curl -s 'https://amazon.com/robots.txt' | sed -n '273,274p'curl -s 'https://amazon.com/robots.txt' | sed -n '246,247p' | ||||||||
| ↳ | #US product | Crawler operators | aws.amazon.com | Open | — | AWS CloudFront / WAF | Open | — |
|
No AI crawler is blocked. AWS CloudFront / WAF serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 4 | Crawler operators | anthropic.com | Open | — | Cloudflare | Open | — | |
|
No AI crawler is blocked. Cloudflare serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| ↳ | #US product | Crawler operators | claude.ai | robots.txt blocks searchOAI-SearchBot, ChatGPT-User | OpenAI, Google | Cloudflare | MixedGPTBot, Google-Extended | Invalid |
|
Cloudflare's bot protection lets verified crawlers in (Claude, Google, Bing, Apple and 5 more), but PerplexityBot, Perplexity-User, Amzn-User and Google-GeminiNotebook are not on Cloudflare's verified list, so they get the same challenge as our test and can't read the site. Tested from 2 vantage points (2 usable) · robots.txt readable · AI tokens in robots.txt: GPTBot, OAI-SearchBot, ChatGPT-User, Google-Extended · llms.txt present · checked 6 November 2026 OAI-SearchBot robots.txtClaude-SearchBot Allowed (verified)PerplexityBot Not verifiedmeta-webindexer Allowed (verified)Amzn-SearchBot Allowed (verified)MistralAI-Index Allowed (verified)DuckAssistBot Allowed (verified)YouBot Allowed (verified)Googlebot Allowed (verified)bingbot Allowed (verified)Applebot Allowed (verified)ChatGPT-User robots.txtClaude-User Allowed (verified)Perplexity-User Not verifiedmeta-externalfetcher Allowed (verified)MistralAI-User Allowed (verified)Amzn-User Not verifiedGoogle-Agent Allowed (verified)Google-GeminiNotebook Not verifiedGPTBot robots.txtClaudeBot Allowed (verified)CCBot Not verifiedmeta-externalagent Allowed (verified)Amazonbot Allowed (verified)MistralAI-Training Not verifiedAI2Bot Not verifiedGoogle-CloudVertexBot Allowed (verified)Diffbot Not verifiedTimpibot Not verifiedwebzio Not verifiedImagesiftBot Not verifiedKagibot Allowed (verified)GoogleOther Allowed (verified)Google-Extended robots.txtApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
Show evidence (4 commands: every verdict other than a plain allow)OAI-SearchBot · Disallowed in robots.txt · line 10: Disallow: / · robots.txt:curl -s 'https://claude.ai/robots.txt' | sed -n '6,10p'curl -s 'https://claude.ai/robots.txt' | sed -n '6,10p'curl -s 'https://claude.ai/robots.txt' | sed -n '6,10p'curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; [email protected])' 'https://claude.ai/login' | ||||||||
| ↳ | #US product | Crawler operators | claude.com | Open | — | Cloudflare | Open | — |
|
No AI crawler is blocked. Cloudflare serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 5 | Other AI companies | baidu.com | robots.txt blocks searchOAI-SearchBot, Claude-SearchBot, PerplexityBot, meta-webindexer, Amzn-SearchBot, MistralAI-Index +7 more | OpenAI, Anthropic, Perplexity, Meta +14Disallow: / for everyone | No CDN / WAF detected | RestrictedGPTBot, ClaudeBot, CCBot, meta-externalagent… | — | |
|
Robots.txt disallows 29 crawlers; nothing blocks them at the edge. Tested from 2 vantage points (2 usable) · robots.txt readable · checked 6 November 2026 OAI-SearchBot robots.txtClaude-SearchBot robots.txtPerplexityBot robots.txtmeta-webindexer robots.txtAmzn-SearchBot robots.txtMistralAI-Index robots.txtDuckAssistBot robots.txtYouBot robots.txtGooglebot Allowedbingbot robots.txtApplebot robots.txtChatGPT-User robots.txtClaude-User robots.txtPerplexity-User Allowedmeta-externalfetcher robots.txtMistralAI-User robots.txtAmzn-User robots.txtGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot robots.txtClaudeBot robots.txtCCBot robots.txtmeta-externalagent robots.txtAmazonbot robots.txtMistralAI-Training robots.txtAI2Bot robots.txtGoogle-CloudVertexBot robots.txtDiffbot robots.txtTimpibot robots.txtwebzio robots.txtImagesiftBot robots.txtKagibot robots.txtGoogleOther robots.txtGoogle-Extended robots.txtApplebot-Extended robots.txtwebzio-extended robots.txtCoherebot robots.txt
Show evidence (8 commands: every verdict other than a plain allow)OAI-SearchBot · Disallowed in robots.txt · line 165: Disallow: / · robots.txt:curl -s 'https://baidu.com/robots.txt' | sed -n '164,165p'curl -s 'https://baidu.com/robots.txt' | sed -n '164,165p'curl -s 'https://baidu.com/robots.txt' | sed -n '164,165p'curl -s 'https://baidu.com/robots.txt' | sed -n '164,165p'curl -s 'https://baidu.com/robots.txt' | sed -n '164,165p'curl -s 'https://baidu.com/robots.txt' | sed -n '164,165p'curl -s 'https://baidu.com/robots.txt' | sed -n '164,165p'curl -s 'https://baidu.com/robots.txt' | sed -n '164,165p' | ||||||||
| 6 | Crawler operators | meta.com | Blocks assistantsGoogle-GeminiNotebook | No CDN / WAF detected | Open | — | ||
|
1 crawler did not get the page while a browser and our honest control did: Google-GeminiNotebook got HTTP 400 instead of the page. Tested from 2 vantage points (1 usable) · robots.txt readable · silent blocks: Google-GeminiNotebook · checked 7 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook Blocked (named)GPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
Show evidence (1 command: every verdict other than a plain allow)Google-GeminiNotebook · Blocked by name · HTTP 400 · http 400:curl -sSI -A 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36 (compatible; Google-GeminiNotebook; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-gemininotebook)' 'https://www.meta.com/gb/' | ||||||||
| ↳ | #US product | Crawler operators | ai.meta.com | Blocks assistantsGoogle-GeminiNotebook | No CDN / WAF detected | Open | — | |
|
1 crawler did not get the page while a browser and our honest control did: Google-GeminiNotebook got HTTP 400 instead of the page. Tested from 2 vantage points (2 usable) · robots.txt readable · silent blocks: Google-GeminiNotebook · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook Blocked (named)GPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot InconsistentKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
Show evidence (1 command: every verdict other than a plain allow)Google-GeminiNotebook · Blocked by name · HTTP 400 · http 400:curl -sSI -A 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36 (compatible; Google-GeminiNotebook; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-gemininotebook)' 'https://ai.meta.com/' | ||||||||
| ↳ | #US product | Crawler operators | meta.ai | robots.txt blocks searchOAI-SearchBot, Claude-SearchBot, PerplexityBot, meta-webindexer, Amzn-SearchBot, MistralAI-Index +7 more | Perplexity, OpenAI, Anthropic, Amazon +14Disallow: / for everyone | No CDN / WAF detected | RestrictedGPTBot, ClaudeBot, CCBot, meta-externalagent… | — |
|
The site lets 5 crawler user agents through (Googlebot, bingbot, Applebot, Google-CloudVertexBot and 1 more) and refuses every other automated client, our honest bot included: even a browser gets HTTP 403. 3 AI search crawlers and assistants are not on that list and get the same HTTP 403 (Perplexity-User, Google-Agent and Google-GeminiNotebook). Tested from 2 vantage points (2 usable) · robots.txt readable · AI tokens in robots.txt: Amazonbot, Applebot-Extended, ClaudeBot, Google-Extended, GPTBot, PerplexityBot · checked 6 November 2026 OAI-SearchBot robots.txtClaude-SearchBot robots.txtPerplexityBot robots.txtmeta-webindexer robots.txtAmzn-SearchBot robots.txtMistralAI-Index robots.txtDuckAssistBot robots.txtYouBot robots.txtGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User robots.txtClaude-User robots.txtPerplexity-User Bot mgmtmeta-externalfetcher robots.txtMistralAI-User robots.txtAmzn-User robots.txtGoogle-Agent Bot mgmtGoogle-GeminiNotebook Bot mgmtGPTBot robots.txtClaudeBot robots.txtCCBot robots.txtmeta-externalagent robots.txtAmazonbot robots.txtMistralAI-Training robots.txtAI2Bot robots.txtGoogle-CloudVertexBot robots.txtDiffbot robots.txtTimpibot robots.txtwebzio robots.txtImagesiftBot robots.txtKagibot robots.txtGoogleOther robots.txtGoogle-Extended robots.txtApplebot-Extended robots.txtwebzio-extended robots.txtCoherebot robots.txt
Show evidence (8 commands: every verdict other than a plain allow)OAI-SearchBot · Disallowed in robots.txt · line 52: Disallow: / · robots.txt:curl -s 'https://meta.ai/robots.txt' | sed -n '51,52p'curl -s 'https://meta.ai/robots.txt' | sed -n '51,52p'curl -s 'https://meta.ai/robots.txt' | sed -n '8,22p'curl -s 'https://meta.ai/robots.txt' | sed -n '51,52p'curl -s 'https://meta.ai/robots.txt' | sed -n '51,52p'curl -s 'https://meta.ai/robots.txt' | sed -n '51,52p'curl -s 'https://meta.ai/robots.txt' | sed -n '51,52p'curl -s 'https://meta.ai/robots.txt' | sed -n '51,52p' | ||||||||
| 7 | Forbes AI 50 | midjourney.com | robots.txt blocks searchYouBot | You.com, OpenAI, Common Crawl, Meta +5 | Cloudflare | MixedGPTBot, CCBot, meta-externalagent, Amazonbot… | — | |
|
Cloudflare's bot protection lets verified crawlers in (ChatGPT, Claude, Google, Bing and 5 more), but PerplexityBot, Perplexity-User, Amzn-User and Google-GeminiNotebook are not on Cloudflare's verified list, so they get the same challenge as our test and can't read the site. Tested from 2 vantage points (2 usable) · robots.txt readable · AI tokens in robots.txt: GPTBot, CCBot, anthropic-ai, Bytespider, FacebookBot, Diffbot, Omgilibot, Omgili, ImagesiftBot, cohere-ai, cohere-training-data-crawler, Amazonbot, YouBot, Google-Extended, Applebot-Extended, meta-externalagent · retired names: anthropic-ai, FacebookBot, Omgilibot, Omgili · checked 6 November 2026 OAI-SearchBot Allowed (verified)Claude-SearchBot Allowed (verified)PerplexityBot Not verifiedmeta-webindexer Allowed (verified)Amzn-SearchBot Allowed (verified)MistralAI-Index Allowed (verified)DuckAssistBot Allowed (verified)YouBot robots.txtGooglebot Allowed (verified)bingbot Allowed (verified)Applebot Allowed (verified)ChatGPT-User Allowed (verified)Claude-User Allowed (verified)Perplexity-User Not verifiedmeta-externalfetcher Allowed (verified)MistralAI-User Allowed (verified)Amzn-User Not verifiedGoogle-Agent Allowed (verified)Google-GeminiNotebook Not verifiedGPTBot robots.txtClaudeBot Allowed (verified)CCBot robots.txtmeta-externalagent robots.txtAmazonbot robots.txtMistralAI-Training Not verifiedAI2Bot Not verifiedGoogle-CloudVertexBot Allowed (verified)Diffbot robots.txtTimpibot Not verifiedwebzio Not verifiedImagesiftBot robots.txtKagibot Allowed (verified)GoogleOther Allowed (verified)Google-Extended robots.txtApplebot-Extended robots.txtwebzio-extended AllowedCoherebot Allowed
Show evidence (8 commands: every verdict other than a plain allow)YouBot · Disallowed in robots.txt · line 97: Disallow: / · robots.txt:curl -s 'https://midjourney.com/robots.txt' | sed -n '96,97p'curl -s 'https://midjourney.com/robots.txt' | sed -n '54,55p'curl -s 'https://midjourney.com/robots.txt' | sed -n '57,58p'curl -s 'https://midjourney.com/robots.txt' | sed -n '111,112p'curl -s 'https://midjourney.com/robots.txt' | sed -n '92,93p'curl -s 'https://midjourney.com/robots.txt' | sed -n '69,70p'curl -s 'https://midjourney.com/robots.txt' | sed -n '78,79p'curl -sSI -A 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot' 'https://www.midjourney.com/' | ||||||||
| 8 | Forbes AI 50 | notion.com | robots.txt blocks searchAmzn-SearchBot | Amazon, Meta, Imagesift | Cloudflare | MixedImagesiftBot, meta-externalagent, Amazonbot | Yes | |
|
Robots.txt disallows 1 AI search crawler and assistant (Amzn-SearchBot); Cloudflare also blocks 1 AI training crawler at the edge (ImagesiftBot). Tested from 2 vantage points (2 usable) · robots.txt readable · AI tokens in robots.txt: Amazonbot, Amzn-SearchBot, meta-externalagent · silent blocks: ImagesiftBot · llms.txt valid · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot robots.txtMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent robots.txtAmazonbot robots.txtMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot Blocked (named)Kagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
Show evidence (4 commands: every verdict other than a plain allow)Amzn-SearchBot · Disallowed in robots.txt · line 24: Disallow: / · robots.txt:curl -s 'https://notion.com/robots.txt' | sed -n '23,24p'curl -s 'https://notion.com/robots.txt' | sed -n '26,27p'curl -s 'https://notion.com/robots.txt' | sed -n '20,21p'curl -sSI -A 'Mozilla/5.0 (compatible; ImagesiftBot; +imagesift.com)' 'https://www.notion.com/' | ||||||||
| 9 | Crawler operators | openai.com | Verified bots onlynot verified: PerplexityBot, Perplexity-User, Amzn-User… | — | Cloudflare | Open | — | |
|
Cloudflare's bot protection lets verified crawlers in (ChatGPT, Claude, Google, Bing and 6 more), but PerplexityBot, Perplexity-User, Amzn-User and Google-GeminiNotebook are not on Cloudflare's verified list, so they get the same challenge as our test and can't read the site. Tested from 2 vantage points (2 usable) · robots.txt readable · checked 6 November 2026 OAI-SearchBot Allowed (verified)Claude-SearchBot Allowed (verified)PerplexityBot Not verifiedmeta-webindexer Allowed (verified)Amzn-SearchBot Allowed (verified)MistralAI-Index AllowedDuckAssistBot Allowed (verified)YouBot Allowed (verified)Googlebot Allowed (verified)bingbot Allowed (verified)Applebot Allowed (verified)ChatGPT-User Allowed (verified)Claude-User Allowed (verified)Perplexity-User Not verifiedmeta-externalfetcher Allowed (verified)MistralAI-User Allowed (verified)Amzn-User Not verifiedGoogle-Agent Allowed (verified)Google-GeminiNotebook Not verifiedGPTBot Allowed (verified)ClaudeBot Allowed (verified)CCBot Not verifiedmeta-externalagent Allowed (verified)Amazonbot Allowed (verified)MistralAI-Training Not verifiedAI2Bot Not verifiedGoogle-CloudVertexBot Allowed (verified)Diffbot Not verifiedTimpibot Not verifiedwebzio Not verifiedImagesiftBot Not verifiedKagibot AllowedGoogleOther Allowed (verified)Google-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
Show evidence (4 commands: every verdict other than a plain allow)PerplexityBot · Refused: not on Cloudflare's verified list (inferred) · HTTP 403 · line 2: Allow: / · cf-mitigated: challenge:curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)' 'https://openai.com/'curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)' 'https://openai.com/'curl -sSI -A 'CCBot/2.0 (https://commoncrawl.org/faq/)' 'https://openai.com/'curl -sSI -A 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot' 'https://openai.com/' | ||||||||
| ↳ | #US product | Crawler operators | chatgpt.com | robots.txt blocks searchPerplexityBot | Perplexity, Common Crawl, Google | Cloudflare | MixedCCBot, Google-Extended | — |
|
Robots.txt disallows 2 AI crawlers (PerplexityBot and CCBot); nothing blocks them at the edge. Tested from 2 vantage points (2 usable) · robots.txt readable · AI tokens in robots.txt: CCBot, Google-Extended, anthropic-ai, Claude-Web, Omgilibot, Omgili, FacebookBot, Bytespider, PerplexityBot, Perplexity‑User · retired names: anthropic-ai, Claude-Web, Omgilibot, Omgili, FacebookBot · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot robots.txtmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot robots.txtmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended robots.txtApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
Show evidence (2 commands: every verdict other than a plain allow)PerplexityBot · Disallowed in robots.txt · line 40: Disallow: / · robots.txt:curl -s 'https://chatgpt.com/robots.txt' | sed -n '39,40p'curl -s 'https://chatgpt.com/robots.txt' | sed -n '3,4p' | ||||||||
| 10 | Other AI companies | poe.com | robots.txt blocks searchOAI-SearchBot, Claude-SearchBot, PerplexityBot, meta-webindexer, Amzn-SearchBot, MistralAI-Index +7 more | OpenAI, Anthropic, Perplexity, Meta +14Disallow: / for everyone | Cloudflare | MixedClaudeBot, CCBot, meta-externalagent, Amazonbot… | Invalid | |
|
Robots.txt disallows 14 AI search crawlers and assistants (OAI-SearchBot, Claude-SearchBot, PerplexityBot, meta-webindexer and 10 more). Behind it, Cloudflare's bot protection lets verified crawlers in but locks out Perplexity-User and Google-GeminiNotebook (not Cloudflare-verified). Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt present · checked 6 November 2026 OAI-SearchBot robots.txtClaude-SearchBot robots.txtPerplexityBot robots.txtmeta-webindexer robots.txtAmzn-SearchBot robots.txtMistralAI-Index robots.txtDuckAssistBot robots.txtYouBot robots.txtGooglebot Allowed (verified)bingbot Allowed (verified)Applebot robots.txtChatGPT-User robots.txtClaude-User robots.txtPerplexity-User Not verifiedmeta-externalfetcher robots.txtMistralAI-User robots.txtAmzn-User robots.txtGoogle-Agent Allowed (verified)Google-GeminiNotebook Not verifiedGPTBot Allowed (verified)ClaudeBot robots.txtCCBot robots.txtmeta-externalagent robots.txtAmazonbot robots.txtMistralAI-Training robots.txtAI2Bot robots.txtGoogle-CloudVertexBot robots.txtDiffbot robots.txtTimpibot robots.txtwebzio robots.txtImagesiftBot robots.txtKagibot robots.txtGoogleOther robots.txtGoogle-Extended AllowedApplebot-Extended robots.txtwebzio-extended robots.txtCoherebot robots.txt
Show evidence (9 commands: every verdict other than a plain allow)OAI-SearchBot · Disallowed in robots.txt · line 34: Disallow: / · robots.txt:curl -s 'https://poe.com/robots.txt' | sed -n '19,34p'curl -s 'https://poe.com/robots.txt' | sed -n '19,34p'curl -s 'https://poe.com/robots.txt' | sed -n '19,34p'curl -s 'https://poe.com/robots.txt' | sed -n '19,34p'curl -s 'https://poe.com/robots.txt' | sed -n '19,34p'curl -s 'https://poe.com/robots.txt' | sed -n '19,34p'curl -s 'https://poe.com/robots.txt' | sed -n '19,34p'curl -s 'https://poe.com/robots.txt' | sed -n '19,34p'curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36' 'https://poe.com/login?redirect_url=%2F' | ||||||||
| 11 | Other AI companies | tencent.com | Blocks assistantsGoogle-Agent, Google-GeminiNotebook | Google, OpenAI, Anthropic, Common Crawl +9 | Origin (nginx) | MixedGPTBot, ClaudeBot, CCBot, meta-externalagent… | — | |
|
Nginx serves 16 crawler user agents a different page (an HTTP 200 response unlike the page a browser and our honest control get): a soft block. Tested from 2 vantage points (2 usable) · robots.txt readable · silent blocks: Google-Agent, Google-GeminiNotebook, GPTBot, ClaudeBot, CCBot, meta-externalagent, Amazonbot, MistralAI-Training, AI2Bot, Google-CloudVertexBot, Diffbot, Timpibot, webzio, ImagesiftBot, Kagibot, GoogleOther · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent Blocked (named)Google-GeminiNotebook Blocked (named)GPTBot Blocked (named)ClaudeBot Blocked (named)CCBot Blocked (named)meta-externalagent Blocked (named)Amazonbot Blocked (named)MistralAI-Training Blocked (named)AI2Bot Blocked (named)Google-CloudVertexBot Blocked (named)Diffbot Blocked (named)Timpibot Blocked (named)webzio Blocked (named)ImagesiftBot Blocked (named)Kagibot Blocked (named)GoogleOther Blocked (named)Google-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
Show evidence (8 commands: every verdict other than a plain allow)Google-Agent · Blocked by name · HTTP 200 · ok:curl -s -A 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Google-Agent; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-agent) Chrome/W.X.Y.Z Safari/537.36' 'https://www.tencent.com/' | grep -o '<title>[^<]*'curl -s -A 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36 (compatible; Google-GeminiNotebook; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-gemininotebook)' 'https://www.tencent.com/' | grep -o '<title>[^<]*'curl -s -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot' 'https://www.tencent.com/' | grep -o '<title>[^<]*'curl -s -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; [email protected])' 'https://www.tencent.com/' | grep -o '<title>[^<]*'curl -s -A 'CCBot/2.0 (https://commoncrawl.org/faq/)' 'https://www.tencent.com/' | grep -o '<title>[^<]*'curl -s -A 'meta-externalagent/1.1' 'https://www.tencent.com/' | grep -o '<title>[^<]*'curl -s -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1) Chrome/W.X.Y.Z Safari/537.36' 'https://www.tencent.com/' | grep -o '<title>[^<]*'curl -s -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-Training/1.0; +https://docs.mistral.ai/robots)' 'https://www.tencent.com/' | grep -o '<title>[^<]*' | ||||||||
| 12 | Crawler operators | microsoft.com | Training only | Webz.io, Google | Azure Front Door | Mixedwebzio, GoogleOther | — | |
|
No AI search crawler or assistant is blocked. webzio and GoogleOther (training crawlers) got a different page (HTTP 200): a policy choice that costs no AI search visibility. Tested from 2 vantage points (2 usable) · robots.txt readable · silent blocks: webzio, GoogleOther · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio Blocked (named)ImagesiftBot AllowedKagibot AllowedGoogleOther Blocked (named)Google-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
Show evidence (2 commands: every verdict other than a plain allow)webzio · Blocked by name · HTTP 200 · ok:curl -s -A 'webzio' 'https://www.microsoft.com/es-es' | grep -o '<title>[^<]*'curl -s -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GoogleOther) Chrome/W.X.Y.Z Safari/537.36' 'https://www.microsoft.com/es-es' | grep -o '<title>[^<]*' | ||||||||
| ↳ | #US product | Crawler operators | bing.com | Open | — | No CDN / WAF detected | Open | — |
|
No AI crawler is blocked. The site serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| ↳ | #US product | Crawler operators | copilot.microsoft.com | Open | — | Azure Front Door | Open | Invalid |
|
No AI crawler is blocked. Azure Front Door serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt present · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 13 | Crawler operators | mistral.ai | Open | — | Cloudflare | Open | Yes | |
|
No AI crawler is blocked. Cloudflare serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt valid · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| ↳ | #FR product | Crawler operators | chat.mistral.ai | Verified bots onlynot verified: PerplexityBot, Perplexity-User, Amzn-User… | OpenAI, Google | Cloudflare | MixedGPTBot, Google-Extended | Invalid |
|
Cloudflare's bot protection lets verified crawlers in (ChatGPT, Claude, Google, Apple and 5 more), but PerplexityBot, Perplexity-User, Amzn-User and Google-GeminiNotebook are not on Cloudflare's verified list, so they get the same challenge as our test and can't read the site. Tested from 2 vantage points (2 usable) · robots.txt readable · AI tokens in robots.txt: GPTBot, Google-Extended · llms.txt present · checked 6 November 2026 OAI-SearchBot Allowed (verified)Claude-SearchBot Allowed (verified)PerplexityBot Not verifiedmeta-webindexer Allowed (verified)Amzn-SearchBot Allowed (verified)MistralAI-Index Allowed (verified)DuckAssistBot Allowed (verified)YouBot Allowed (verified)Googlebot Allowed (verified)bingbot Allowed (verified)Applebot Allowed (verified)ChatGPT-User Allowed (verified)Claude-User Allowed (verified)Perplexity-User Not verifiedmeta-externalfetcher Allowed (verified)MistralAI-User Allowed (verified)Amzn-User Not verifiedGoogle-Agent Allowed (verified)Google-GeminiNotebook Not verifiedGPTBot robots.txtClaudeBot Allowed (verified)CCBot Not verifiedmeta-externalagent Allowed (verified)Amazonbot Allowed (verified)MistralAI-Training Not verifiedAI2Bot Not verifiedGoogle-CloudVertexBot Allowed (verified)Diffbot Not verifiedTimpibot Not verifiedwebzio Not verifiedImagesiftBot Not verifiedKagibot Allowed (verified)GoogleOther Allowed (verified)Google-Extended robots.txtApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
Show evidence (5 commands: every verdict other than a plain allow)GPTBot · Disallowed in robots.txt · line 2: Disallow: / · robots.txt:curl -s 'https://chat.mistral.ai/robots.txt' | sed -n '1,2p'curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)' 'https://chat.mistral.ai/chat'curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)' 'https://chat.mistral.ai/chat'curl -sSI -A 'CCBot/2.0 (https://commoncrawl.org/faq/)' 'https://chat.mistral.ai/chat'curl -sSI -A 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot' 'https://chat.mistral.ai/chat' | ||||||||
| 14 | Forbes AI 50 | reflection.ai | Training only | OpenAI, Anthropic, Common Crawl, Meta +5 | Cloudflare | MixedGPTBot, ClaudeBot, CCBot, meta-externalagent… | — | |
|
Robots.txt disallows 9 AI crawlers; nothing blocks them at the edge. Tested from 2 vantage points (2 usable) · robots.txt readable · AI tokens in robots.txt: Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Diffbot, Google-Extended, GPTBot, omgili, anthropic-ai, Claude-Web, cohere-ai, MistralAI-Training, meta-externalagent, GoogleOther, Google-CloudVertexBot · retired names: omgili, anthropic-ai, Claude-Web · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot robots.txtClaudeBot robots.txtCCBot robots.txtmeta-externalagent robots.txtAmazonbot robots.txtMistralAI-Training robots.txtAI2Bot AllowedGoogle-CloudVertexBot robots.txtDiffbot robots.txtTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther robots.txtGoogle-Extended robots.txtApplebot-Extended robots.txtwebzio-extended AllowedCoherebot Allowed
Show evidence (8 commands: every verdict other than a plain allow)GPTBot · Disallowed in robots.txt · line 57: Disallow: / · robots.txt:curl -s 'https://reflection.ai/robots.txt' | sed -n '56,57p'curl -s 'https://reflection.ai/robots.txt' | sed -n '47,48p'curl -s 'https://reflection.ai/robots.txt' | sed -n '44,45p'curl -s 'https://reflection.ai/robots.txt' | sed -n '74,75p'curl -s 'https://reflection.ai/robots.txt' | sed -n '35,36p'curl -s 'https://reflection.ai/robots.txt' | sed -n '71,72p'curl -s 'https://reflection.ai/robots.txt' | sed -n '90,91p'curl -s 'https://reflection.ai/robots.txt' | sed -n '50,51p' | ||||||||
| 15 | Forbes AI 50 | together.ai | Training only | Cloudflare | MixedGoogle-Extended | Yes | ||
|
No AI crawler is blocked. Cloudflare serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · AI tokens in robots.txt: Google-Extended · llms.txt valid · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended robots.txtApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 16 | Forbes AI 50 | x.ai | Verified bots onlynot verified: PerplexityBot, Perplexity-User, Google-GeminiNotebook… | — | Cloudflare | Open | — | |
|
Cloudflare's bot protection lets verified crawlers in (ChatGPT, Claude, Google, Bing and 5 more), but PerplexityBot, Perplexity-User and Google-GeminiNotebook are not on Cloudflare's verified list, so they get the same refusal as our test and can't read the site. Tested from 2 vantage points (2 usable) · robots.txt readable · checked 6 November 2026 OAI-SearchBot Allowed (verified)Claude-SearchBot Allowed (verified)PerplexityBot Not verifiedmeta-webindexer Allowed (verified)Amzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot Allowed (verified)YouBot Allowed (verified)Googlebot Allowed (verified)bingbot Allowed (verified)Applebot Allowed (verified)ChatGPT-User Allowed (verified)Claude-User Allowed (verified)Perplexity-User Not verifiedmeta-externalfetcher Allowed (verified)MistralAI-User Allowed (verified)Amzn-User AllowedGoogle-Agent Allowed (verified)Google-GeminiNotebook Not verifiedGPTBot Allowed (verified)ClaudeBot Allowed (verified)CCBot Not verifiedmeta-externalagent Allowed (verified)Amazonbot Allowed (verified)MistralAI-Training AllowedAI2Bot Not verifiedGoogle-CloudVertexBot Allowed (verified)Diffbot AllowedTimpibot Not verifiedwebzio AllowedImagesiftBot Not verifiedKagibot AllowedGoogleOther Allowed (verified)Google-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
Show evidence (4 commands: every verdict other than a plain allow)PerplexityBot · Refused: not on Cloudflare's verified list (inferred) · HTTP 403 · line 2: Allow: / · title "Attention Required! | Cloudflare":curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)' 'https://x.ai/'curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)' 'https://x.ai/'curl -sSI -A 'CCBot/2.0 (https://commoncrawl.org/faq/)' 'https://x.ai/'curl -sSI -A 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot' 'https://x.ai/' | ||||||||
| ↳ | #US product | Forbes AI 50 | grok.com | Training only | OpenAI, Anthropic, Common Crawl, Amazon | Cloudflare | MixedGPTBot, ClaudeBot, CCBot, Amazonbot | — |
|
Cloudflare blocks 4 AI training crawlers at the edge (GPTBot, ClaudeBot, CCBot and Amazonbot): a policy choice that keeps content out of model training. No AI search crawler or assistant is blocked. Tested from 2 vantage points (2 usable) · robots.txt readable · AI tokens in robots.txt: GPTBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended, Applebot-Extended · silent blocks: GPTBot, ClaudeBot, CCBot, Amazonbot · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot Blocked (named)ClaudeBot Blocked (named)CCBot Blocked (named)meta-externalagent AllowedAmazonbot Blocked (named)MistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
Show evidence (4 commands: every verdict other than a plain allow)GPTBot · Blocked by name · HTTP 403 · line 20: Allow: /$ · body: "Your request was blocked." (text/plain — Cloudflare AI Crawl Control policy response):curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot' 'https://grok.com/'curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; [email protected])' 'https://grok.com/'curl -sSI -A 'CCBot/2.0 (https://commoncrawl.org/faq/)' 'https://grok.com/'curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1) Chrome/W.X.Y.Z Safari/537.36' 'https://grok.com/' | ||||||||
| 17 | Other AI companies | character.ai | Verified bots onlynot verified: PerplexityBot, Perplexity-User, Amzn-User… | — | Cloudflare | Open | — | |
|
Cloudflare's bot protection lets verified crawlers in (ChatGPT, Claude, Google, Bing and 6 more), but PerplexityBot, Perplexity-User, Amzn-User and Google-GeminiNotebook are not on Cloudflare's verified list, so they get the same challenge as our test and can't read the site. Tested from 2 vantage points (2 usable) · robots.txt readable · checked 6 November 2026 OAI-SearchBot Allowed (verified)Claude-SearchBot Allowed (verified)PerplexityBot Not verifiedmeta-webindexer Allowed (verified)Amzn-SearchBot Allowed (verified)MistralAI-Index Allowed (verified)DuckAssistBot Allowed (verified)YouBot Allowed (verified)Googlebot Allowed (verified)bingbot Allowed (verified)Applebot Allowed (verified)ChatGPT-User Allowed (verified)Claude-User Allowed (verified)Perplexity-User Not verifiedmeta-externalfetcher Allowed (verified)MistralAI-User Allowed (verified)Amzn-User Not verifiedGoogle-Agent Allowed (verified)Google-GeminiNotebook Not verifiedGPTBot Allowed (verified)ClaudeBot Allowed (verified)CCBot Not verifiedmeta-externalagent Allowed (verified)Amazonbot Allowed (verified)MistralAI-Training Not verifiedAI2Bot Not verifiedGoogle-CloudVertexBot Allowed (verified)Diffbot Not verifiedTimpibot Not verifiedwebzio Not verifiedImagesiftBot Not verifiedKagibot Allowed (verified)GoogleOther Allowed (verified)Google-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
Show evidence (4 commands: every verdict other than a plain allow)PerplexityBot · Refused: not on Cloudflare's verified list (inferred) · HTTP 403 · title "Attention Required! | Cloudflare":curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)' 'https://character.ai/'curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)' 'https://character.ai/'curl -sSI -A 'CCBot/2.0 (https://commoncrawl.org/faq/)' 'https://character.ai/'curl -sSI -A 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot' 'https://character.ai/' | ||||||||
| 18 | Forbes AI 50 | databricks.com | Verified bots onlynot verified: Google-GeminiNotebook | — | Cloudflare | Open | Yes | |
|
Cloudflare's bot protection lets verified crawlers in (Google, Bing), but Google-GeminiNotebook is not on Cloudflare's verified list, so it gets the same refusal as our test and can't read the site. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt valid · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowed (verified)bingbot Allowed (verified)Applebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent Allowed (verified)Google-GeminiNotebook Not verifiedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 19 | Forbes AI 50 | gamma.app | Verified bots onlynot verified: PerplexityBot, Perplexity-User, Amzn-User… | — | Cloudflare | Open | Yes | |
|
Cloudflare's bot protection lets verified crawlers in (ChatGPT, Claude, Google, Bing and 6 more), but PerplexityBot, Perplexity-User, Amzn-User and Google-GeminiNotebook are not on Cloudflare's verified list, so they get the same challenge as our test and can't read the site. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt valid · checked 6 November 2026 OAI-SearchBot Allowed (verified)Claude-SearchBot Allowed (verified)PerplexityBot Not verifiedmeta-webindexer Allowed (verified)Amzn-SearchBot Allowed (verified)MistralAI-Index Allowed (verified)DuckAssistBot Allowed (verified)YouBot Allowed (verified)Googlebot Allowed (verified)bingbot Allowed (verified)Applebot Allowed (verified)ChatGPT-User Allowed (verified)Claude-User Allowed (verified)Perplexity-User Not verifiedmeta-externalfetcher Allowed (verified)MistralAI-User Allowed (verified)Amzn-User Not verifiedGoogle-Agent Allowed (verified)Google-GeminiNotebook Not verifiedGPTBot Allowed (verified)ClaudeBot Allowed (verified)CCBot Not verifiedmeta-externalagent Allowed (verified)Amazonbot Allowed (verified)MistralAI-Training Not verifiedAI2Bot Not verifiedGoogle-CloudVertexBot Allowed (verified)Diffbot Not verifiedTimpibot Not verifiedwebzio Not verifiedImagesiftBot Not verifiedKagibot Allowed (verified)GoogleOther Allowed (verified)Google-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
Show evidence (4 commands: every verdict other than a plain allow)PerplexityBot · Refused: not on Cloudflare's verified list (inferred) · HTTP 403 · cf-mitigated: challenge:curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)' 'https://gamma.app/'curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)' 'https://gamma.app/'curl -sSI -A 'CCBot/2.0 (https://commoncrawl.org/faq/)' 'https://gamma.app/'curl -sSI -A 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot' 'https://gamma.app/' | ||||||||
| 20 | Forbes AI 50 | genspark.ai | Verified bots onlynot verified: PerplexityBot, Perplexity-User, Google-GeminiNotebook… | — | Cloudflare | Open | Yes | |
|
Cloudflare's bot protection lets verified crawlers in (ChatGPT, Claude, Google, Bing and 5 more), but PerplexityBot, Perplexity-User and Google-GeminiNotebook are not on Cloudflare's verified list, so they get the same challenge as our test and can't read the site. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt valid · checked 6 November 2026 OAI-SearchBot Allowed (verified)Claude-SearchBot Allowed (verified)PerplexityBot Not verifiedmeta-webindexer Allowed (verified)Amzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot Allowed (verified)YouBot Allowed (verified)Googlebot Allowed (verified)bingbot Allowed (verified)Applebot Allowed (verified)ChatGPT-User Allowed (verified)Claude-User Allowed (verified)Perplexity-User Not verifiedmeta-externalfetcher Allowed (verified)MistralAI-User Allowed (verified)Amzn-User AllowedGoogle-Agent Allowed (verified)Google-GeminiNotebook Not verifiedGPTBot Allowed (verified)ClaudeBot Allowed (verified)CCBot Not verifiedmeta-externalagent Allowed (verified)Amazonbot Allowed (verified)MistralAI-Training AllowedAI2Bot Not verifiedGoogle-CloudVertexBot Allowed (verified)Diffbot AllowedTimpibot Not verifiedwebzio AllowedImagesiftBot Not verifiedKagibot AllowedGoogleOther Allowed (verified)Google-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
Show evidence (4 commands: every verdict other than a plain allow)PerplexityBot · Refused: not on Cloudflare's verified list (inferred) · HTTP 403 · cf-mitigated: challenge:curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)' 'https://www.genspark.ai/'curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)' 'https://www.genspark.ai/'curl -sSI -A 'CCBot/2.0 (https://commoncrawl.org/faq/)' 'https://www.genspark.ai/'curl -sSI -A 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot' 'https://www.genspark.ai/' | ||||||||
| 21 | Forbes AI 50 | lovable.dev | Verified bots onlynot verified: PerplexityBot, Perplexity-User, Amzn-User… | — | Cloudflare | Open | Yes | |
|
Cloudflare's bot protection lets verified crawlers in (ChatGPT, Claude, Google, Bing and 6 more), but PerplexityBot, Perplexity-User, Amzn-User and Google-GeminiNotebook are not on Cloudflare's verified list, so they get the same challenge as our test and can't read the site. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt valid · checked 6 November 2026 OAI-SearchBot Allowed (verified)Claude-SearchBot Allowed (verified)PerplexityBot Not verifiedmeta-webindexer Allowed (verified)Amzn-SearchBot Allowed (verified)MistralAI-Index Allowed (verified)DuckAssistBot Allowed (verified)YouBot Allowed (verified)Googlebot Allowed (verified)bingbot Allowed (verified)Applebot Allowed (verified)ChatGPT-User Allowed (verified)Claude-User Allowed (verified)Perplexity-User Not verifiedmeta-externalfetcher Allowed (verified)MistralAI-User Allowed (verified)Amzn-User Not verifiedGoogle-Agent Allowed (verified)Google-GeminiNotebook Not verifiedGPTBot Allowed (verified)ClaudeBot Allowed (verified)CCBot Not verifiedmeta-externalagent Allowed (verified)Amazonbot Allowed (verified)MistralAI-Training Not verifiedAI2Bot Not verifiedGoogle-CloudVertexBot Allowed (verified)Diffbot Not verifiedTimpibot Not verifiedwebzio Not verifiedImagesiftBot Not verifiedKagibot Allowed (verified)GoogleOther Allowed (verified)Google-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
Show evidence (4 commands: every verdict other than a plain allow)PerplexityBot · Refused: not on Cloudflare's verified list (inferred) · HTTP 403 · line 56: Allow: / · cf-mitigated: challenge:curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)' 'https://lovable.dev/'curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)' 'https://lovable.dev/'curl -sSI -A 'CCBot/2.0 (https://commoncrawl.org/faq/)' 'https://lovable.dev/'curl -sSI -A 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot' 'https://lovable.dev/' | ||||||||
| 22 | Other AI companies | deepseek.com | Open | — | AWS CloudFront / WAF | Open | — | |
|
No AI crawler is blocked. AWS CloudFront / WAF serves every tested user agent the same page as a browser. Tested from 2 vantage points (1 usable) · robots.txt readable · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot Not testedChatGPT-User Not testedClaude-User Not testedPerplexity-User Not testedmeta-externalfetcher Not testedMistralAI-User Not testedAmzn-User Not testedGoogle-Agent Not testedGoogle-GeminiNotebook Not testedGPTBot Not testedClaudeBot Not testedCCBot Not testedmeta-externalagent Not testedAmazonbot Not testedMistralAI-Training Not testedAI2Bot Not testedGoogle-CloudVertexBot Not testedDiffbot Not testedTimpibot Not testedwebzio Not testedImagesiftBot Not testedKagibot Not testedGoogleOther Not testedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| ↳ | #CN product | Other AI companies | chat.deepseek.com | Verification WAF | — | AWS CloudFront / WAF | Open | — |
|
AWS CloudFront / WAF refuses requests that claim a known crawler identity without verifying it. A spoofed Googlebot or bingbot is blocked the same way, so it is checking who crawlers are, not blocking AI by name. Tested from 2 vantage points (1 usable) · robots.txt unreadable challenge · checked 6 November 2026 OAI-SearchBot UnverifiedClaude-SearchBot ChallengedPerplexityBot Unverifiedmeta-webindexer UnverifiedAmzn-SearchBot ChallengedMistralAI-Index UnverifiedDuckAssistBot UnverifiedYouBot UnverifiedGooglebot Unverifiedbingbot UnverifiedApplebot UnverifiedChatGPT-User UnverifiedClaude-User UnverifiedPerplexity-User Unverifiedmeta-externalfetcher UnverifiedMistralAI-User UnverifiedAmzn-User ChallengedGoogle-Agent UnverifiedGoogle-GeminiNotebook UnverifiedGPTBot UnverifiedClaudeBot UnverifiedCCBot Unverifiedmeta-externalagent UnverifiedAmazonbot UnverifiedMistralAI-Training UnverifiedAI2Bot UnverifiedGoogle-CloudVertexBot UnverifiedDiffbot UnverifiedTimpibot Unverifiedwebzio UnverifiedImagesiftBot UnverifiedKagibot UnverifiedGoogleOther ChallengedGoogle-Extended Not testedApplebot-Extended Not testedwebzio-extended Not testedCoherebot Not tested
Show evidence (8 commands: every verdict other than a plain allow)OAI-SearchBot · Refused as unverified · HTTP 403 · http 403:curl -sSI -A 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot' 'https://chat.deepseek.com/'curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; [email protected])' 'https://chat.deepseek.com/'curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)' 'https://chat.deepseek.com/'curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36' 'https://chat.deepseek.com/'curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot' 'https://chat.deepseek.com/'curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; [email protected])' 'https://chat.deepseek.com/'curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)' 'https://chat.deepseek.com/'curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot' 'https://chat.deepseek.com/' | ||||||||
| 23 | Forbes AI 50 | openevidence.com | Bot management | — | Vercel | Open | — | |
|
Vercel refuses non-browser clients generally. Our honest, self-identifying control got the same HTTP 403 as 33 AI crawlers while a browser passes, so this is bot management, not a rule naming them. Tested from 2 vantage points (2 usable) · robots.txt readable · checked 6 November 2026 OAI-SearchBot Bot mgmtClaude-SearchBot Bot mgmtPerplexityBot Bot mgmtmeta-webindexer Bot mgmtAmzn-SearchBot Bot mgmtMistralAI-Index Bot mgmtDuckAssistBot Bot mgmtYouBot Bot mgmtGooglebot Bot mgmtbingbot Bot mgmtApplebot Bot mgmtChatGPT-User Bot mgmtClaude-User Bot mgmtPerplexity-User Bot mgmtmeta-externalfetcher Bot mgmtMistralAI-User Bot mgmtAmzn-User Bot mgmtGoogle-Agent Bot mgmtGoogle-GeminiNotebook Bot mgmtGPTBot Bot mgmtClaudeBot Bot mgmtCCBot Bot mgmtmeta-externalagent Bot mgmtAmazonbot Bot mgmtMistralAI-Training Bot mgmtAI2Bot Bot mgmtGoogle-CloudVertexBot Bot mgmtDiffbot Bot mgmtTimpibot Bot mgmtwebzio Bot mgmtImagesiftBot Bot mgmtKagibot Bot mgmtGoogleOther Bot mgmtGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
Show evidence (8 commands: every verdict other than a plain allow)OAI-SearchBot · Refused as unrecognised bot · HTTP 403 · x-vercel-mitigated: deny:curl -sSI -A 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot' 'https://www.openevidence.com/'curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; [email protected])' 'https://www.openevidence.com/'curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)' 'https://www.openevidence.com/'curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36' 'https://www.openevidence.com/'curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot' 'https://www.openevidence.com/'curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; [email protected])' 'https://www.openevidence.com/'curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)' 'https://www.openevidence.com/'curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot' 'https://www.openevidence.com/' | ||||||||
| 24 | Forbes AI 50 | replit.com | Challenged | — | Cloudflare | Open | Yes | |
|
Cloudflare's bot protection lets verified crawlers in (Google, Bing); it only refuses unverified copies like our test, so it costs you no AI visibility. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt valid · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowed (verified)bingbot Allowed (verified)Applebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot ChallengedDiffbot AllowedTimpibot Challengedwebzio AllowedImagesiftBot ChallengedKagibot AllowedGoogleOther ChallengedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 25 | Other AI companies | ai21.com | Open | — | Cloudflare | Open | Yes | |
|
No AI crawler is blocked. Cloudflare serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt valid · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 26 | Forbes AI 50 | abridge.com | Open | — | Cloudflare | Open | — | |
|
No AI crawler is blocked. Cloudflare serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 27 | Crawler operators | allenai.org | Open | — | Google Cloud | Open | — | |
|
No AI crawler is blocked. Google Cloud serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 28 | Other AI companies | aleph-alpha.com | Open | — | Origin (nginx) | Open | — | |
|
No AI crawler is blocked. Nginx serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 29 | Other AI companies | qwen.ai | Open | — | No CDN / WAF detected | Open | Invalid | |
|
No AI crawler is blocked. The site serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt not text · llms.txt present · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended Not testedApplebot-Extended Not testedwebzio-extended Not testedCoherebot Not tested
| ||||||||
| 30 | Other AI companies | alibabacloud.com | Open | — | No CDN / WAF detected | Open | Invalid | |
|
No AI crawler is blocked. The site serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt present · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 31 | Crawler operators | apple.com | Open | — | No CDN / WAF detected | Open | — | |
|
No AI crawler is blocked. The site serves every tested user agent the same page as a browser. Tested from 2 vantage points (1 usable) · robots.txt readable · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 32 | Forbes AI 50 | appliedintuition.com | Open | — | Vercel | Open | Yes | |
|
No AI crawler is blocked. Vercel serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt valid · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 33 | Forbes AI 50 | baseten.co | Open | — | Vercel | Open | Yes | |
|
No AI crawler is blocked. Vercel serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt valid · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 34 | Forbes AI 50 | bfl.ai | Open | — | Vercel | Open | Yes | |
|
No AI crawler is blocked. Vercel serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt valid · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 35 | Crawler operators | brave.com | Open | — | AWS CloudFront / WAF | Open | — | |
|
No AI crawler is blocked. AWS CloudFront / WAF serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| ↳ | #US product | Crawler operators | search.brave.com | Open | — | AWS CloudFront / WAF | Open | — |
|
No AI crawler is blocked. AWS CloudFront / WAF serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent Not testedGoogle-GeminiNotebook Not testedGPTBot Not testedClaudeBot Not testedCCBot Not testedmeta-externalagent Not testedAmazonbot Not testedMistralAI-Training Not testedAI2Bot Not testedGoogle-CloudVertexBot Not testedDiffbot Not testedTimpibot Not testedwebzio Not testedImagesiftBot Not testedKagibot Not testedGoogleOther Not testedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 36 | Crawler operators | bytedance.com | Open | — | No CDN / WAF detected | Open | — | |
|
No AI crawler is blocked. The site serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 37 | Other AI companies | cerebras.ai | Open | — | Vercel | Open | — | |
|
No AI crawler is blocked. Vercel serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 38 | Forbes AI 50 | chaidiscovery.com | Open | — | No CDN / WAF detected | Open | — | |
|
No AI crawler is blocked. The site serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 39 | Forbes AI 50 | clay.com | Open | — | Cloudflare | Open | — | |
|
No AI crawler is blocked. Cloudflare serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 40 | Forbes AI 50 | cognition.com | Open | — | Vercel | Open | Yes | |
|
No AI crawler is blocked. Vercel serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt valid · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| ↳ | #US product | Forbes AI 50 | devin.ai | Open | — | Vercel | Open | Yes |
|
No AI crawler is blocked. Vercel serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt valid · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 41 | Forbes AI 50 | cohere.com | Open | — | Vercel | Open | Yes | |
|
No AI crawler is blocked. Vercel serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt valid · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 42 | Crawler operators | commoncrawl.org | Open | — | Cloudflare | Open | — | |
|
No AI crawler is blocked. Cloudflare serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 43 | Forbes AI 50 | crusoe.ai | Open | — | Cloudflare | Open | Yes | |
|
No AI crawler is blocked. Cloudflare serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt valid · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 44 | Forbes AI 50 | cursor.com | Open | — | Vercel | Open | Yes | |
|
No AI crawler is blocked. Vercel serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt valid · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 45 | Forbes AI 50 | cyera.com | Open | — | Cloudflare | Open | Yes | |
|
No AI crawler is blocked. Cloudflare serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt valid · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 46 | Forbes AI 50 | decagon.ai | Open | — | Cloudflare | Open | Yes | |
|
No AI crawler is blocked. Cloudflare serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt valid · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 47 | Crawler operators | diffbot.com | Open | — | Origin (nginx) | Open | Yes | |
|
No AI crawler is blocked. Nginx serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt valid · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 48 | Crawler operators | duckduckgo.com | Open | — | Origin (nginx) | Open | Invalid | |
|
No AI crawler is blocked. Nginx serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt present · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 49 | Forbes AI 50 | elevenlabs.io | Open | — | Google Cloud | Open | Yes | |
|
No AI crawler is blocked. Google Cloud serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt valid · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 50 | Forbes AI 50 | eliseai.com | Open | — | Cloudflare | Open | Invalid | |
|
No AI crawler is blocked. Cloudflare serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt present · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 51 | Forbes AI 50 | fal.ai | Open | — | Cloudflare | Open | Yes | |
|
No AI crawler is blocked. Cloudflare serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt valid · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 52 | Forbes AI 50 | fireworks.ai | Open | — | Vercel | Open | Yes | |
|
No AI crawler is blocked. Vercel serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt valid · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 53 | Forbes AI 50 | glean.com | Open | — | Cloudflare | Open | — | |
|
No AI crawler is blocked. Cloudflare serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 54 | Crawler operators | google.com | Open | — | Google Cloud | Open | — | |
|
No AI crawler is blocked. Google Cloud serves every tested user agent the same page as a browser. Tested from 2 vantage points (1 usable) · robots.txt readable · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| ↳ | #US product | Crawler operators | gemini.google.com | Open | — | No CDN / WAF detected | Open | — |
|
No AI crawler is blocked. The site serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 55 | Crawler operators | deepmind.google | Open | — | Google Cloud | Open | Yes | |
|
No AI crawler is blocked. Google Cloud serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt valid · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 56 | Other AI companies | grammarly.com | Open | — | AWS CloudFront / WAF | Open | — | |
|
No AI crawler is blocked. AWS CloudFront / WAF serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 57 | Other AI companies | groq.com | Open | — | Vercel | Open | — | |
|
No AI crawler is blocked. Vercel serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 58 | Forbes AI 50 | harvey.ai | Open | — | Vercel | Open | — | |
|
No AI crawler is blocked. Vercel serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 59 | Forbes AI 50 | heygen.com | Open | — | Cloudflare | Open | Yes | |
|
No AI crawler is blocked. Cloudflare serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt valid · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 60 | Crawler operators | thehive.ai | Open | — | AWS CloudFront / WAF | Open | Yes | |
|
No AI crawler is blocked. AWS CloudFront / WAF serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt valid · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 61 | Other AI companies | huggingface.co | Open | — | AWS CloudFront / WAF | Open | — | |
|
No AI crawler is blocked. AWS CloudFront / WAF serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 62 | Other AI companies | ideogram.ai | Open | — | Cloudflare | Open | Yes | |
|
No AI crawler is blocked. Cloudflare serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt valid · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 63 | Other AI companies | inflection.ai | Open | — | No CDN / WAF detected | Open | — | |
|
No AI crawler is blocked. The site serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 64 | Other AI companies | jasper.ai | Open | — | Cloudflare | Open | Yes | |
|
No AI crawler is blocked. Cloudflare serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · AI tokens in robots.txt: Omgilibot · retired names: Omgilibot · llms.txt valid · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 65 | Crawler operators | kagi.com | Open | — | Google Cloud | Open | — | |
|
No AI crawler is blocked. Google Cloud serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 66 | Forbes AI 50 | krea.ai | Open | — | Cloudflare | Open | Yes | |
|
No AI crawler is blocked. Cloudflare serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt valid · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 67 | Forbes AI 50 | legora.com | Open | — | No CDN / WAF detected | Open | Yes | |
|
No AI crawler is blocked. The site serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt valid · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 68 | Other AI companies | liquid.ai | Open | — | Vercel | Open | — | |
|
No AI crawler is blocked. Vercel serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 69 | Forbes AI 50 | listenlabs.ai | Open | — | Vercel | Open | Yes | |
|
No AI crawler is blocked. Vercel serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt valid · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 70 | Other AI companies | lumalabs.ai | Open | — | Vercel | Open | Yes | |
|
No AI crawler is blocked. Vercel serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt valid · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 71 | Other AI companies | manus.im | Open | — | AWS CloudFront / WAF | Open | — | |
|
No AI crawler is blocked. AWS CloudFront / WAF serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 72 | Forbes AI 50 | mercor.com | Open | — | Cloudflare | Open | Yes | |
|
No AI crawler is blocked. Cloudflare serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt valid · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 73 | Other AI companies | minimax.io | Open | — | Akamai | Open | Invalid | |
|
No AI crawler is blocked. Akamai serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt present · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 74 | Other AI companies | moonshot.cn | Open | — | Origin (nginx) | Open | Invalid | |
|
No AI crawler is blocked. Nginx serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt present · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| ↳ | #CN product | Other AI companies | kimi.com | Open | — | Cloudflare | Open | Invalid |
|
No AI crawler is blocked. Cloudflare serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt present · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 75 | Other AI companies | nvidia.com | Open | — | Origin (Apache) | Open | Yes | |
|
No AI crawler is blocked. Apache serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt valid · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 76 | Other AI companies | otter.ai | Open | — | Cloudflare | Open | Yes | |
|
No AI crawler is blocked. Cloudflare serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt valid · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 77 | Forbes AI 50 | pi.website | Open | — | Vercel | Open | — | |
|
No AI crawler is blocked. Vercel serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 78 | Other AI companies | pika.art | Open | — | Vercel | Open | Yes | |
|
No AI crawler is blocked. Vercel serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt valid · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 79 | Other AI companies | replika.com | Open | — | No CDN / WAF detected | Open | Yes | |
|
No AI crawler is blocked. The site serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt valid · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 80 | Forbes AI 50 | rogo.com | Open | — | Vercel | Open | Yes | |
|
No AI crawler is blocked. Vercel serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt valid · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 81 | Forbes AI 50 | runway.com | Open | — | Vercel | Open | Yes | |
|
No AI crawler is blocked. Vercel serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt valid · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 82 | Forbes AI 50 | ssi.inc | Open | — | Vercel | Open | — | |
|
No AI crawler is blocked. Vercel serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt not found · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended Not testedApplebot-Extended Not testedwebzio-extended Not testedCoherebot Not tested
| ||||||||
| 83 | Other AI companies | sakana.ai | Open | — | Fastly | Open | — | |
|
No AI crawler is blocked. Fastly serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt not found · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended Not testedApplebot-Extended Not testedwebzio-extended Not testedCoherebot Not tested
| ||||||||
| 84 | Forbes AI 50 | sambanova.ai | Open | — | Cloudflare | Open | Yes | |
|
No AI crawler is blocked. Cloudflare serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt valid · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 85 | Other AI companies | scale.com | Open | — | Vercel | Open | — | |
|
No AI crawler is blocked. Vercel serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 86 | Forbes AI 50 | sierra.ai | Open | — | Cloudflare | Open | — | |
|
No AI crawler is blocked. Cloudflare serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 87 | Forbes AI 50 | skild.ai | Open | — | Cloudflare | Open | — | |
|
No AI crawler is blocked. Cloudflare serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 88 | Forbes AI 50 | speak.com | Open | — | Cloudflare | Open | Yes | |
|
No AI crawler is blocked. Cloudflare serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt valid · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 89 | Other AI companies | stability.ai | Open | — | Squarespace | Open | — | |
|
No AI crawler is blocked. Squarespace serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 90 | Forbes AI 50 | suno.com | Open | — | Cloudflare | Open | Yes | |
|
No AI crawler is blocked. Cloudflare serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt valid · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 91 | Forbes AI 50 | surgehq.ai | Open | — | Cloudflare | Open | — | |
|
No AI crawler is blocked. Cloudflare serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 92 | Forbes AI 50 | synthesia.io | Open | — | Cloudflare | Open | Yes | |
|
No AI crawler is blocked. Cloudflare serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt valid · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 93 | Forbes AI 50 | thinkingmachines.ai | Open | — | Cloudflare | Open | — | |
|
No AI crawler is blocked. Cloudflare serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 94 | Crawler operators | timpi.com | Open | — | Origin (Apache) | Open | Invalid | |
|
No AI crawler is blocked. Apache serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt not text · llms.txt present · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended Not testedApplebot-Extended Not testedwebzio-extended Not testedCoherebot Not tested
| ||||||||
| 95 | Other AI companies | udio.com | Open | — | Vercel | Open | — | |
|
No AI crawler is blocked. Vercel serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 96 | Crawler operators | webz.io | Open | — | Cloudflare | Open | Yes | |
|
No AI crawler is blocked. Cloudflare serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt valid · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 97 | Forbes AI 50 | worldlabs.ai | Open | — | Vercel | Open | — | |
|
No AI crawler is blocked. Vercel serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 98 | Crawler operators | you.com | Open | — | Cloudflare | Open | Yes | |
|
No AI crawler is blocked. Cloudflare serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt valid · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| 99 | Other AI companies | zhipuai.cn | Open | — | No CDN / WAF detected | Open | Invalid | |
|
No AI crawler is blocked. The site serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt present · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
| ↳ | #CN product | Other AI companies | chat.z.ai | Open | — | No CDN / WAF detected | Open | Invalid |
|
No AI crawler is blocked. The site serves every tested user agent the same page as a browser. Tested from 2 vantage points (2 usable) · robots.txt readable · llms.txt present · checked 6 November 2026 OAI-SearchBot AllowedClaude-SearchBot AllowedPerplexityBot Allowedmeta-webindexer AllowedAmzn-SearchBot AllowedMistralAI-Index AllowedDuckAssistBot AllowedYouBot AllowedGooglebot Allowedbingbot AllowedApplebot AllowedChatGPT-User AllowedClaude-User AllowedPerplexity-User Allowedmeta-externalfetcher AllowedMistralAI-User AllowedAmzn-User AllowedGoogle-Agent AllowedGoogle-GeminiNotebook AllowedGPTBot AllowedClaudeBot AllowedCCBot Allowedmeta-externalagent AllowedAmazonbot AllowedMistralAI-Training AllowedAI2Bot AllowedGoogle-CloudVertexBot AllowedDiffbot AllowedTimpibot Allowedwebzio AllowedImagesiftBot AllowedKagibot AllowedGoogleOther AllowedGoogle-Extended AllowedApplebot-Extended Allowedwebzio-extended AllowedCoherebot Allowed
| ||||||||
Showing the first 3 companies. Filtering, sorting or a row link opens the whole table.
Paste your domain into the free AI Visibility Checker: it runs the same 33-crawler test, names the layer that blocks each crawler and gives you the fix.
Everything on this page may be quoted and reused with credit under CC BY 4.0. The dataset, charts and per-company breakdowns are free to download, and press requests get a same-day answer.
Cite as: LinksThatRank, AI Companies AI Visibility Report, November 2026, https://linksthatrank.com/ai-visibility-checker/which-ai-companies-block-ai-crawlers/
Everything a reporter, editor or technical reader needs to check exactly how each classification was made. The full method is on the methodology page.
* group applies to crawlers not named, with the rule and its line number recorded for every disallow. Retired crawler names (anthropic-ai, Claude-Web, FacebookBot, Omgili) are reported as such: they match no current crawler. Google-Extended and Applebot-Extended are robots.txt-only opt-out tokens, read from the file and never sent as user agents.checker_score_best_vantage for reference only.The test requests deliberately claimed crawler identities without the private network or verification signals authentic crawlers can provide. On a site running Cloudflare’s verified-bot gate, the observed fact is that our copy of the crawler and a spoofed Googlebot got the same challenge. The verdict for crawlers not on Cloudflare’s published list follows from that list, and the report labels it as an inference everywhere it appears.
When a site also refuses our invented, never-seen bot the same way it refuses an AI crawler, the evidence points to generic bot management rather than an AI-specific policy. Only a rule that names the crawler - in robots.txt or at the edge - is reported as a block, and training-crawler restrictions never count against a company in this report.