About GEO Search Console

Common Crawl is a nonprofit that has archived the public web since 2008 and gives the data away free.

It runs a new crawl about once a month, each holding a few billion pages, and its bot is called CCBot. Many AI training datasets start from these crawls, which is why site owners now ask whether their pages are in them. Common Crawl's own site has more on the project, and our guide to what Common Crawl is covers how it works.

The tools are free, and GEO Search Console isn't affiliated with Anthropic, OpenAI, Google, Exa, Brave or the Common Crawl Foundation.

There's nothing to sign up for. The checker reads Common Crawl's public URL index and archive, data anyone can query, and puts the answer in plain words: saved, not in this crawl, redirected, or blocked, along with what did the blocking. The Common Crawl checker guide walks through each result.

AI Visibility and Fan-Out pay outside services for every answer, so each connection gets a set number of free checks a day, more for visitors who sign in with Google, and the site has a daily total. Bulk lists, for visitors signed in with Google, have their own daily allowance and their own daily total, so a long list never uses up the free checks everyone else shares.

Knowledge Panel asks Google's free Knowledge Graph Search API, which has a daily quota, so each connection gets 40 searches a day (120 when signed in with Google). A name anyone searched in the last 24 hours comes back from our copy and doesn't count.

For privacy, Crawl Record and AI Performance keep nothing, because they run in your browser. AI Visibility, Fan-Out and Knowledge Panel run on our server and keep a little, for a short time.

When you press Check in Crawl Record, your browser asks Common Crawl's public index and archive directly, and the answers come straight back to your screen. Crawl Record has no accounts, and it keeps no list of what people check.

AI Performance doesn't even go that far: the exports you drop in are read by your browser, the report is built there, and nothing in them is sent anywhere. Close or reload the tab and they're gone.

1. Your browserasksindex.commoncrawl.orgWhich crawls hold the address, when, and with what HTTP status.
2. Your browserasksdata.commoncrawl.orgOne saved record when needed: the robots.txt the bot read, or the page it was shown.
3. Your browserasksthe site you checkedOnly when you press "Show the page as saved" on a saved copy: its images and styles load from the live site.

You can watch all three in your browser's developer tools, on the Network tab.

AI Visibility and Fan-Out work differently. Our server sends the address, search or prompt you type to the services that answer it: DataForSEO (for the ChatGPT, Claude, Gemini and Google results) and Brave Search. AI Visibility also requests the page you check and its robots.txt, once as a normal browser and once with each AI crawler's name.

Knowledge Panel works the same way. Our server sends the name or ID you type to Google's Knowledge Graph Search API and, when Google links an entity to a Wikipedia article, asks Wikipedia for that article's picture. The pictures then load in your browser straight from Wikimedia's servers.

Each answer is kept for 24 hours so a repeat check doesn't cost twice, and then it expires. Brave Search results are the exception: Brave's terms allow only short-term use, so we don't keep them. To share the free checks fairly, we count checks per connection under a scrambled code made from your connection's address, never the address itself, and delete those counts after a few days. If you sign in, we count them under your account instead.

Signing in with Google and weekly reports are the only things that store an email address. Asking for a weekly report sends a confirmation email, and nothing more arrives until you open its link. Every report has a link that stops it, and stopping deletes the address. Sign-ups that are never confirmed are deleted after 7 days.

Signing in is optional. It raises your daily checks and opens bulk lists. When you sign in with Google, Google tells us your email address, your name and an ID number for your account. We keep those, plus when you signed up and last visited. We never see your Google password, and we can't read or post anything in your Google account. A cookie keeps you signed in for up to 90 days. Sign out ends it, and Delete my sign-in, on either tool page, removes all of it.

Before a check that costs us money, visitors who aren't signed in pass a quick check from Cloudflare Turnstile that they're a person. Most of the time it needs no click. Cloudflare sees that check, and a cookie on this site remembers for 12 hours that you passed. That cookie holds no personal details.

Common Crawl's servers see each lookup the way any website sees a visit. Like any website, this one runs on a web host whose server logs record the addresses people open. A share link carries the address you checked in its URL, so opening one shows up in the logs like any other visit. A check started from the home page doesn't: the address travels after a # in the link, and browsers never send that part to the server. The addresses, searches and prompts that AI Visibility, Fan-Out and Knowledge Panel send to our server do show up in those logs. Fonts load from Google Fonts, which sees a request for the font files and nothing you type.

The site uses Google Analytics to count visits. It sets cookies so it can tell a new visitor from a returning one, and Google sees which page was opened and where the visit came from. It never gets the addresses, searches or prompts you type: the site cuts them out of what it sends. Any tracker blocker turns Analytics off, and every tool still works.

To contact GEO Search Console, email info@geosearchconsole.co.

Questions about a result are easiest to answer with the share link from the checker, so we see the same check you did.