Search Check a prompt

Log file analyzer: Which bots read your site?

This free log file analyzer reads your server log in your browser and shows every search and AI bot that visited, whether each one is real or fake, and which pages ChatGPT, Perplexity and Claude fetched to answer someone.

Drop your log files here

Access logs as they download: cPanel Raw Access (.gz), Apache, nginx, Cloudflare Logpush, IIS or a CSV. Add more than one to cover more days.

Free, no sign-up. Your log is read in this browser and never uploaded. Our server only sends the bots' published address lists, and reads your sitemap when you type your site.

How the log file analyzer works

The log file analyzer reads your access log line by line in your browser, keeps only the requests whose user agent names a bot, and checks each one against the addresses its owner publishes.

How to check if Googlebot is crawling your site

To check if Googlebot is crawling your site, look for Googlebot in your access log and make sure each request comes from one of the addresses Google publishes.

Is ChatGPT, Claude or Perplexity visiting your site?

Is ChatGPT, Claude or Perplexity visiting your site? Your log answers it with two kinds of visit: crawlers that collect pages ahead of time, and answer fetches made while someone waits for a reply.

AI crawlers

GPTBot, ClaudeBot and CCBot collect pages that can be used to train models. OAI-SearchBot, Claude-SearchBot and PerplexityBot build the search indexes that ChatGPT, Claude and Perplexity look in. Each company says its robots.txt rules apply to these.

AI answer fetches

ChatGPT-User, Claude-User and Perplexity-User read a page because a person asked something and the AI went to your page for the answer. Each one is close to a citation, so the analyzer lists them by page.

Seeing none?

Check that your robots.txt and firewall let these bots in with AI Visibility. Some hosts and firewalls turn AI crawlers away with 403 or 429 codes, which show up in the broken list.

Where to find your server log files

Where to find your server log files depends on your host, but most give you the access log as a download.

How to tell a fake Googlebot from the real one

To tell a fake Googlebot from the real one, check the address it came from: anyone can write Googlebot in a user agent, but only Google sends requests from Google's published address ranges.

Why bots skip some of your pages

Why bots skip some of your pages usually comes down to how they find them: links, the sitemap and how often the page changes.

Crawl budget: is yours wasted?

Crawl budget is the number of pages a search engine will crawl on your site in a given time, and is wasted when bots spend their visits on errors, redirects and pages you don't need found.

Log file analyzer questions

These log file analyzer questions come up most: what gets uploaded, which logs work and what the labels mean.

What is log file analysis?

Log file analysis is reading your server's access log to see who asked for which pages, when, and what your server answered. For SEO it shows what search engines and AI bots really fetch, which no crawler or rank tool can see from the outside.

Is my log uploaded anywhere?

No. The log is read in your browser and stays on your computer. Our server only sends the page the bots' published address lists, and reads your sitemap if you type your site. Your log lines, addresses and paths are never sent.

Which log formats does the log file analyzer read?

Apache and nginx access logs in the common and combined formats, cPanel Raw Access files (plain or .gz, archives with many days included), Cloudflare Logpush JSON lines and other JSON logs, IIS logs in the W3C format, and CSV files with a header that names the user agent.

Why is a bot marked Fake?

The request names a bot, such as Googlebot or GPTBot, whose owner publishes the addresses it uses, and the request came from an address that isn't on that list. Scrapers often borrow a well-known bot's name.

What is the difference between GPTBot and ChatGPT-User?

GPTBot is OpenAI's crawler for pages that can be used to train its models. ChatGPT-User fetches a page when a ChatGPT user's question leads ChatGPT to read it. OAI-SearchBot builds the index ChatGPT search looks in.

How big a log can it read?

Logs of hundreds of megabytes read in under a minute on a recent computer. A .gz file is opened whole, so very large archives need more memory: split them by month if your browser stops.

Why are people's visits left out?

The analyzer counts only requests whose user agent says it is a bot or a crawler. Visits from browsers are left out, so nothing about your visitors shows in the report.