AI visibility checker: Can AI search find your page?
This free AI visibility checker asks three sources that feed AI answers about your page: Claude's search, ChatGPT and Common Crawl. Type a page address to see which can fetch it, which have it in their search, and what your robots.txt tells each one.
Many pages? Run a bulk list and download one CSV with a row for each page. Needs a Google sign-in.
- Can it fetch the page?
- The source successfully retrieved and extracted content for that request. Where we ask as the bot, our server, using the bot's name, got this answer. A firewall that checks where a request really comes from can treat the real bot differently.
- Is the page in its search?
- The page appeared in those specific search results.
- Do robots.txt rules let it in?
- Whether the site's declared rules permit the crawler.
- History
- Your own observed visibility history, not a confirmed indexing timestamp.
Track this page
Track this page to get a weekly email that repeats these tests and shows what changed.
Bulk check: a list of pages
Paste a list of page addresses, one per line. Every page gets the same sources and tests as a single check, and one CSV holds a row for each page, ready for Google Sheets or Excel.
What each test proves
Each test proves one thing, and none proves more than it says. Together they show where a page gets stuck on its way into AI answers, and the AI visibility audit puts them in order with the other checks.
Can it fetch the page?
The source successfully retrieved and extracted content for that request.
It doesn't prove the page is in that source's search. For Claude, ChatGPT and Common Crawl we send the request from our own server using each bot's name, so a firewall that checks where a request really comes from can treat the real bot differently.
Is the page in its search?
The page appeared in those specific search results.
It shows the page for the search you gave us, not for every search. A different search can give a different answer.
Do robots.txt rules let it in?
Whether the site's declared rules permit the crawler.
robots.txt is a request, not a lock. A firewall can still turn away a crawler it allows, and a crawler can ignore the file.
History
Your own observed visibility history, not a confirmed indexing timestamp.
It starts the week you track a page. "Seen" means one of our weekly checks saw it, not that the source indexed it that day.
How each AI source reaches your page
Each AI source reaches your page in its own way, so the checker asks each one the way it works.
Claude, through Brave Search
Claude's web search reportedly uses Brave's index. Anthropic hasn't announced that. We ask Brave whether it has your page by its address, by your search, and anywhere on your site, then ask Claude your search through Anthropic's API with web search on and see whether it cites your page.
Brave's crawler doesn't announce itself, so the fetch test uses a normal browser as a stand-in. Brave has no robots.txt name of its own and follows the rules for Googlebot.
ChatGPT
ChatGPT search reads the web through OAI-SearchBot, and ChatGPT-User opens a page when someone asks about it. We ask the ChatGPT app your search with web search on and see whether its answer cites your page. This is the slow check: it can take up to a minute.
GPTBot is OpenAI's training crawler. Blocking it doesn't decide whether OAI-SearchBot can reach your page.
Common Crawl
Common Crawl is a nonprofit archive of the web, and many AI training datasets start from it. Your browser asks its index whether the newest crawl saved your page, and its bot is CCBot. For every crawl since 2008, use Crawl Record.
Why a firewall can block a bot that robots.txt allows
A firewall can block a bot that robots.txt allows because robots.txt is a request and a firewall is a gate.
robots.txt tells well-behaved crawlers where they may go. A firewall, such as the one a CDN or host runs, decides whether a request gets an answer at all, and some hosts turn on AI bot blocking by default. Some firewalls also check where a request really comes from, so they refuse a request that only uses a bot's name and let the real bot through.
That is why the checker says "Allowed by robots.txt. But the server blocked it." when the two disagree. If even a normal browser request from our server is refused, the firewall is turning away our server, so the checker says it can't tell.
How to fix a block
To fix a block, find which layer said no, then allow the bot there.
- A 403 or a challenge page. Allow the bot in the site's firewall or CDN settings, such as its AI bot blocking, Bot Fight Mode and any custom security rules.
- A 429. The server is rate limiting the bot. Ask your host to raise the limit for the bots you want.
- A robots.txt block. Change the Disallow line for that crawler, or give it its own group with an Allow line. Remember that Brave follows the Googlebot group.
- Not in Brave. Brave has its own form for new pages, at search.brave.com/submit-url. We never submit anything for you.
- A noindex tag. Remove the tag or header if you want the page in search results.
- Not in Common Crawl. Its bot finds pages through links and sitemaps, so a new page can take several crawls. Crawl Record shows each crawl.