Cart $0.00
0

CART ITEMS:

0
Subtotal: $0.00
No products in the cart.

AI Visibility Checker: Methodology

This page describes exactly what the AI Visibility Checker does when you paste a URL, how each verdict is reached, and what the tool can and cannot know. Every block it measures can be reproduced with a single curl command, which the report prints next to the evidence.

1. What one check does

A check on the free tool is about 40 small HTTP requests to your site (38 on average across a 50-site sample on 4 October 2026; a study check, which requests every crawler, is about 45–50). They are sent from our own servers in data centres in London (DigitalOcean) and New Jersey (Vultr). When a site refuses every automated request from a data centre, the check is re-run from residential-grade ISP addresses, in the site’s own market first (UK or US), and the report says so. Cloudflare remembers addresses: a site behind Cloudflare that has recently challenged our data-centre addresses — or any Cloudflare site, while Cloudflare is challenging those addresses across many sites — is checked from an ISP address from the start, nothing is sent from the data centre, and the report says that too. Requests go at most four at a time, 300 ms apart plus up to 100 ms of random spacing (a request silent for 2.5 seconds gives up its slot, so up to two more may be open while it hangs; never more than four are being answered). Each request gets 7 seconds (less once a site starts dropping crawlers; at most 4 seconds for optional files), follows at most five redirects and reads at most 64 KiB of a page (256 KiB of robots.txt). The check stops after 20 seconds, extended once to 30 seconds when the requests still open can plausibly finish. We never crawl: robots.txt, the homepage, a deeper page only when robots.txt names a path or you entered one, and the policy files listed below.

  1. Normalise and guard the URL. http/https only, ports 80 and 443, no IP literals, no private or reserved addresses (re-checked on every redirect hop), and a "remove my domain" list (see the about page).
  2. Three control requests to the homepage, and one more at the end. C1 is a real browser request: a current Chrome user agent with Chrome’s own TLS and HTTP/2 fingerprint, because bot managers score the connection as well as the user agent. C2 is an honest, self-identifying bot: Mozilla/5.0 (compatible; LTR-AIVisibilityCheck/1.0; +https://linksthatrank.com/ai-visibility-checker/). C3 is an honest bot whose product token is invented for that check and has never been seen anywhere (Mozilla/5.0 (compatible; KaroviBot/1.0; +https://linksthatrank.com/ai-visibility-checker/), say) — because bot-management networks learn the C2 token over time, C3 is what tells “a rule naming this crawler” from “refuses bots it does not recognise” (it is sent only when C2 gets the page). After the crawler requests, C2 is sent once more: if the site now refuses it, the site changed its mind during the test (behavioural detection or a rate limit) and the blocks seen in between are reported as unconfirmed and the check is re-run from another address. If the site does not answer at all we try the www / bare alternative once.
  3. robots.txt from the final host, read as a browser when the browser control got the page (otherwise as our honest bot, and again as a browser if the bot is refused), parsed with RFC 9309 semantics and Google-compatible product-token matching (longest match wins, allow wins ties, * and $ supported). We keep line numbers, Crawl-delay, Content-Signal, License (RSL), Sitemap lines, and the comment markers that identify Cloudflare-managed blocks (Bot Preference Sync), Squarespace, Shopify and Wix.
  4. One request per crawler user agent to the homepage — the exact strings of 33 crawlers, taken from the operators' own documentation (OpenAI, Anthropic, Google, Microsoft, Apple, Perplexity, Meta, Amazon, Mistral, Common Crawl, DuckDuckGo and others), sent with the minimal headers a crawler sends. GET, not HEAD, because HEAD hides block pages. On the free check, seven smaller training crawlers (MistralAI-Training, AI2Bot, Diffbot, Timpibot, webzio, ImagesiftBot and Kagibot) are read from robots.txt rather than requested; they are never scored. The studies request every crawler.
  5. One deeper page, only when it can show something the homepage cannot — the homepage, when the URL you typed was a deeper page; otherwise, when robots.txt has a rule for a specific path, a page from your sitemap or an internal link that robots.txt allows for all crawlers (never a sign-in, account, basket or search page). It is requested with the honest control plus eight high-value crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot, ChatGPT-User, Claude-User and GPTBot), which catches a crawler allowed on the homepage but disallowed deeper in the site. When robots.txt treats every page alike, the check stays on the homepage.
  6. Policy signals — /llms.txt (validated: many are soft-404 HTML), /.well-known/tdmrep.json, X-Robots-Tag and meta noai / noimageai, TDM-Reservation, Content-Signal and Link: rel="license" headers. Reported as footnotes, never scored.

2. How a verdict is reached

Each crawler's response is compared with the controls and with robots.txt:

VerdictRuleConfidence
Allowedrobots.txt allows the path and the live response is a 200 comparable to the honest control (title / size within tolerance; pre-rendered variants for search engines count as allowed).High
Blocked by robots.txtA Disallow rule matches the tested path for that token (the rule and line are shown). The live result is shown alongside; fetchers that ignore robots.txt are noted.High
Blocked at the edge (user-agent rule)The crawler gets a block signature (Cloudflare "Attention Required" page or the AI Crawl Control policy response, Vercel x-vercel-mitigated: deny, Akamai "Access Denied", AWS WAF, Azure Front Door, Imperva, Sucuri, DataDome, HUMAN, Kasada, F5…) while the honest control gets the page — or, when the controls are only challenged, the crawler is blocked outright ("harder than the controls").High
Allowed (verified crawler, inferred)Cloudflare bot protection: our browser gets the page while our copies of Googlebot, Bingbot and Applebot are challenged or refused. Crawlers in Cloudflare’s verified-bot programme arrive from their operators’ own addresses and normally pass, so they are shown as allowed and marked as inferred.Medium
Locked out (not verified)The same site, for a crawler that is not in Cloudflare’s verified-bot programme (PerplexityBot, for one): it is refused for real. The fix, a skip rule for the crawler’s published addresses, would be invisible to our test, so it counts 0.25 rather than nothing.Medium
Refused as an unrecognised bot (bot management)The crawler and the never-seen C3 control get the same refusal while the known honest control passes. That is bot management refusing what it does not recognise, not a rule naming the crawler; verified crawlers are commonly allow-listed. Counted at half weight.Medium
Refused as unverified (identity verification)The crawler is refused exactly as a spoofed Googlebot and bingbot are refused, with the controls passing. The site checks who crawlers are; the real crawler, arriving from its operator’s published addresses, is expected to pass. Crawlers that publish no addresses cannot pass such a check and stay blocked. Half weight for the verifiable ones.Medium
Challenged (only this user agent)Only this crawler string gets a JavaScript challenge while every control passes. A crawler cannot solve it, but a rule naming the string and bot-management scoring are indistinguishable from outside, and verified crawlers are commonly exempt. Counts 0.5.Medium
Blocked at the origin (user-agent rule)A 403 / 444 / 503 with an Apache, nginx, LiteSpeed, IIS or Wordfence signature and no CDN block marker, while the honest control passes and robots.txt itself is served normally.High
Challenged (generic bot detection)The crawler and the controls get the same challenge (cf-mitigated: challenge, a reCAPTCHA interstitial, Vercel 429, DataDome, PerimeterX…). Verified crawlers from their published IP ranges usually pass; we cannot prove it, so this counts 0.75. A challenge to every client, our browser included, counts 0.5.Medium
Challenged everywhere (Cloudflare)Cloudflare challenges both our controls, the browser included. Verified crawlers normally get through a site set up like this; we could not confirm it from our address, so the report says so first and every crawler verdict on that check is marked as inferred.Medium
Refused by an allow-listOur honest bot is refused while named crawlers, a spoofed search crawler included, get the page: the site lets in a list of known crawlers and refuses everything else. Crawlers refused like our bot are not on that list. Counts 0.5.Medium
Never answers non-browser clientsA browser gets the page while our honest bot’s request hangs until it times out. Every crawler is then tried briefly (3 seconds, no retry); those that hang the same way are reported as dropped by bot management. Counts 0.5.Medium
Could not testBoth controls, the browser included, are refused and no crawler is treated worse than them. We say so, re-run from residential-grade addresses and give no score. Akamai also scores individual addresses, so its sites are testable when a clean address is available; the report names any address that was refused.Low
Paid access (402)HTTP 402, with or without a crawler-price header (Cloudflare pay-per-crawl).High
Dropped / soft blockThe connection is closed or left hanging only for the crawler, a 200 carries an interstitial instead of the page, or the crawler is redirected away. Counts 0.25, or 0.5 when an unknown bot is treated the same way (bot management).Medium
Parked domainA registrar’s parking or for-sale page: there is no website to check yet. Reported as a parked domain, with no score.—
Robots-only tokensGoogle-Extended, Applebot-Extended, webzio-extended and Coherebot are evaluated in robots.txt only and never sent as a user agent.—
Not addressableChatGPT agent, Perplexity Comet and Grok publish no robots.txt token, and Brave Search follows whatever you set for Googlebot: none can be addressed on its own. Shown as information rows.—

Per-IP rate limits (HTTP 429) are backed off and reported as "not tested", never as a per-crawler block. The exception is a server that answers named crawlers 429 every time while everyone else gets the page (some Hostinger, LiteSpeed and nginx setups do this to GPTBot and meta-externalagent): when every such 429 holds on its retry and later requests are still answered, that is a rule for those crawlers and is reported as a soft block for them. An invalid HTTPS certificate (expired, for another host name, self-signed or missing its intermediate) is named in the headline: we read past it to see what the site serves, but most crawlers will not. If the check hits its time limit, the skipped rows say so.

3. The score

The AI Visibility Score is a weighted average over the AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, meta-webindexer, Amzn-SearchBot, MistralAI-Index, DuckAssistBot, YouBot) and AI assistants (ChatGPT-User, Claude-User, Perplexity-User, meta-externalfetcher, MistralAI-User, Amzn-User, Google-Agent, Google-GeminiNotebook), plus Googlebot, Bingbot and Applebot as the crawlers behind AI Overviews / AI Mode, Copilot and Apple Intelligence. Allowed = 1 (a verified crawler inferred to pass counts as allowed); challenged by generic bot detection = 0.75 and any other challenge 0.5; refused by bot management, identity verification or an allow-list = 0.5; dropped or soft-blocked = 0.25 (0.5 under bot management); locked out as an unverified crawler = 0.25; blocked by robots.txt or by a rule that names the crawler, or behind paid access = 0. Crawlers that could not be tested are left out, and when fewer than half of them were tested there is no score. Weights reflect how much each crawler matters for being found and cited; they are published in the registry.

Training crawlers are never scored. GPTBot, ClaudeBot, CCBot, meta-externalagent, Amazonbot, MistralAI-Training, AI2Bot, Diffbot and the rest are reported under "training policy" as open, restricted or mixed. Blocking them is a legitimate choice about your content; it does not change whether AI search can recommend you.

4. Where the block lives

Block and challenge pages are recognised per vendor from their documented signatures (status, a few headers, a body marker) — and, for Cloudflare, from a calibration matrix we captured on a zone we control: per-crawler "Block" toggles and WAF block rules serve the HTML "Attention Required" page; the Search / Agent / Training policies answer 403 text/plain "Your request was blocked."; managed and JS challenges carry cf-mitigated: challenge; robots.txt is exempt from AI Crawl Control blocks; and the same policies are visible to an unverified probe only for the crawlers Cloudflare matches by user agent. That is why the report can say "Training = Block" rather than just "403".

5. What the tool cannot know

  • Our requests are unverified. They come from our servers, not from the operators' IP ranges, and they do not carry Web Bot Auth signatures. Results are exact for user-agent-keyed rules and robots.txt; for rules keyed on verified identity they are an upper bound on blocking. When a site treats every unknown client the same, we say so instead of claiming a block.
  • A 200 for a spoofed search crawler on Cloudflare is not proof of access for policies that key on verified identity; the report words those rows as "not blocked by a user-agent rule".
  • Vantage matters. Some sites refuse all data-centre traffic; we re-run those from residential-grade ISP addresses and label the result. Cloudflare scores addresses with a long memory, so a site that has recently challenged our data-centre addresses is checked from an ISP address from the start, and the report says so. Akamai scores individual addresses too: its sites are testable when a clean address is available, and the report names any address that was refused.
  • Results are a snapshot. Bot-management defaults change (Cloudflare changed its AI defaults in July 2025, July 2026 and September 2026). Results are cached for six hours (an incomplete result for ten minutes) and kept for 30 days; monitoring re-checks weekly.

6. Reproducing a finding

Every row blocked at the edge or by the server prints the command we ran. For example:

curl -sS -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot)" \
  https://example.com/

A 403 with a block page while the same command with a plain browser user agent returns 200 is a user-agent rule. Run it from a home connection as well as from a server: some CDNs answer differently by network.

7. Politeness and opt-out

robots.txt, the homepage and at most one deeper page per check (plus the policy files), four requests in flight, 300 ms spacing plus up to 100 ms, 7 seconds per request, one retry (never after a timeout), back-off on 429, a limit on repeat checks of the same site (everyone else sees the stored result), a six-hour result cache, and an honest control user agent with a contact URL. The about page explains how to remove your domain from testing and what we store (the checked domain and a salted daily hash of the requester's IP, for 30 days).