Log file analyzer: Which bots read your site?
This free log file analyzer reads your server log in your browser and shows every search and AI bot that visited, whether each one is real or fake, and which pages ChatGPT, Perplexity and Claude fetched to answer someone.
Drop your log files here
Access logs as they download: cPanel Raw Access (.gz), Apache, nginx, Cloudflare Logpush, IIS or a CSV. Add more than one to cover more days.
Free, no sign-up. Your log is read in this browser and never uploaded. Our server only sends the bots' published address lists, and reads your sitemap when you type your site.
This browser can't open .gz files: unzip the log first, or use Chrome, Edge, Firefox or Safari.
Bots that visited
Pages ChatGPT, Perplexity and Claude fetched for answers
No answer fetches in these lines. When someone asks ChatGPT, Perplexity or Claude a question and it reads one of your pages to answer, the visit shows here. See which pages the answers cite instead with the Citation Gap tool.
What bots hit that was broken
No errors, blocks or redirects: every bot got a page.
Sitemap pages no bot reached
Get new free tools by email
One email when we add a new tool, and one on the 1st of each month before Google deletes a month of your Search Console data. Nothing else.
You confirm by email first. We keep only your address, and every email has a link that stops them and deletes it.
How the log file analyzer works
The log file analyzer reads your access log line by line in your browser, keeps only the requests whose user agent names a bot, and checks each one against the addresses its owner publishes.
- Read the log. Each line gives the address that asked, the time, the page, the status code and the user agent. Plain and .gz files work, and a cPanel archive with many days in it opens in one go.
- Name the bot. The user agent says which bot it claims to be: a search engine, an AI search crawler, an AI training crawler, an AI answer fetch, an SEO tool or a link preview. Requests from people are left out.
- Check it's real. Google, Bing, OpenAI, Anthropic, Perplexity, Apple, Common Crawl and DuckDuckGo publish the addresses their bots use. A request that names one of their bots from another address is marked Fake.
- Find the gaps. Type your site and it reads your sitemap from our server, then lists the pages no bot reached in the days your log covers.
How to check if Googlebot is crawling your site
To check if Googlebot is crawling your site, look for Googlebot in your access log and make sure each request comes from one of the addresses Google publishes.
- In this tool. Drop your log and find Googlebot in the bot table. Verified means every request came from Google's published lists, and the bars show the days it came.
- In Search Console. The Crawl stats report (Settings, then Crawl stats) shows Google's requests, response codes and file types for up to 90 days, without the page-by-page detail a log gives.
- One page at a time. The URL Inspection tool in Search Console says when Google last crawled a page.
Is ChatGPT, Claude or Perplexity visiting your site?
Is ChatGPT, Claude or Perplexity visiting your site? Your log answers it with two kinds of visit: crawlers that collect pages ahead of time, and answer fetches made while someone waits for a reply.
AI crawlers
GPTBot, ClaudeBot and CCBot collect pages that can be used to train models. OAI-SearchBot, Claude-SearchBot and PerplexityBot build the search indexes that ChatGPT, Claude and Perplexity look in. Each company says its robots.txt rules apply to these.
AI answer fetches
ChatGPT-User, Claude-User and Perplexity-User read a page because a person asked something and the AI went to your page for the answer. Each one is close to a citation, so the analyzer lists them by page.
Seeing none?
Check that your robots.txt and firewall let these bots in with AI Visibility. Some hosts and firewalls turn AI crawlers away with 403 or 429 codes, which show up in the broken list.
Where to find your server log files
Where to find your server log files depends on your host, but most give you the access log as a download.
- cPanel. Open Raw Access under Metrics and download the file for your domain. Hosts that keep archives list one .gz file per month.
- Cloudflare. Logpush sends HTTP request logs as JSON lines on Enterprise plans. The analyzer reads the client address, path, status and user agent fields.
- nginx and Apache. On your own server, look in /var/log/nginx/access.log, /var/log/apache2/access.log or /var/log/httpd/access_log.
- IIS. Logs are in inetpub\logs\LogFiles in the W3C format, which the analyzer reads from its #Fields line.
- Managed hosts. Many WordPress and app hosts offer access logs in their dashboard or on request. Ask for the raw access log with user agents.
How to tell a fake Googlebot from the real one
To tell a fake Googlebot from the real one, check the address it came from: anyone can write Googlebot in a user agent, but only Google sends requests from Google's published address ranges.
- Fake. The request names a bot whose owner publishes a list, and its address isn't on that list. Scrapers do this to get past rules written for real crawlers.
- No list. The owner publishes no address list we can check, so the request may or may not be real. Most SEO tools and link previews are in this group.
- Can't check. Claude-User requests in our own log came from many networks and none from Anthropic's published addresses, so the analyzer doesn't call them fake.
- The other way. Google also says you can check a crawler with a reverse DNS lookup of its address and a forward lookup of the name it returns.
Why bots skip some of your pages
Why bots skip some of your pages usually comes down to how they find them: links, the sitemap and how often the page changes.
- No links in. A page with no internal links is hard to find, even when the sitemap lists it.
- Blocked. A robots.txt rule, a firewall or a login keeps bots out. The broken list shows 403s and 429s by bot.
- Not in a sitemap bots read. Make sure robots.txt names your sitemap. Search engines read it, and AI crawlers often follow the links they find.
- Too short a log. A week of log won't show pages bots visit once a month. Add older files to cover more days.
Crawl budget: is yours wasted?
Crawl budget is the number of pages a search engine will crawl on your site in a given time, and is wasted when bots spend their visits on errors, redirects and pages you don't need found.
- Who needs to care. Google says crawl budget matters mostly for very large sites, around a million pages or more, and for sites of 10,000 pages or more that change every day.
- What the log shows. The broken list counts 404s, redirects and server errors by bot, with the pages that got the most.
- What to fix first. Point old addresses straight at their new page, take dead links out of your own pages, and keep error pages from answering 200.
Log file analyzer questions
These log file analyzer questions come up most: what gets uploaded, which logs work and what the labels mean.
What is log file analysis?
Log file analysis is reading your server's access log to see who asked for which pages, when, and what your server answered. For SEO it shows what search engines and AI bots really fetch, which no crawler or rank tool can see from the outside.
Is my log uploaded anywhere?
No. The log is read in your browser and stays on your computer. Our server only sends the page the bots' published address lists, and reads your sitemap if you type your site. Your log lines, addresses and paths are never sent.
Which log formats does the log file analyzer read?
Apache and nginx access logs in the common and combined formats, cPanel Raw Access files (plain or .gz, archives with many days included), Cloudflare Logpush JSON lines and other JSON logs, IIS logs in the W3C format, and CSV files with a header that names the user agent.
Why is a bot marked Fake?
The request names a bot, such as Googlebot or GPTBot, whose owner publishes the addresses it uses, and the request came from an address that isn't on that list. Scrapers often borrow a well-known bot's name.
What is the difference between GPTBot and ChatGPT-User?
GPTBot is OpenAI's crawler for pages that can be used to train its models. ChatGPT-User fetches a page when a ChatGPT user's question leads ChatGPT to read it. OAI-SearchBot builds the index ChatGPT search looks in.
How big a log can it read?
Logs of hundreds of megabytes read in under a minute on a recent computer. A .gz file is opened whole, so very large archives need more memory: split them by month if your browser stops.
Why are people's visits left out?
The analyzer counts only requests whose user agent says it is a bot or a crawler. Visits from browsers are left out, so nothing about your visitors shows in the report.