AI crawlers & AI search
AI assistants such as ChatGPT, Claude or Perplexity read websites with their own crawlers and send more and more visitors. Little of this shows up in regular Matomo tracking, because crawlers do not run JavaScript. I analyse the server logs instead, with my own pipeline based on Filebeat that brings crawler and AI bot hits into Matomo. It is already running for many clients.
How I work
- 01Review server logs and hosting
- 02Set up the pipeline, detect and classify crawlers
- 03Crawler hits as a separate website in Matomo
- 04Set up a channel for visitors from AI assistants
Two questions, two data sources
- Who reads along? AI crawlers and fetchers collect content for training, search and answers. They appear in the server logs, not in JavaScript tracking.
- Who comes back? Visitors who arrive via a link in ChatGPT, Perplexity or other assistants are measured by Matomo as usual. They only need a channel of their own.
The pipeline
Filebeat reads your web server’s access logs, detects crawlers and AI bots and transfers their hits to a separate website in Matomo. There you see per crawler which pages are fetched how often, over time and next to your regular reports.
What it helps you decide
Which content AI services read most often, whether new crawlers appear, whether blocks in robots.txt work, and whether visibility in AI answers shows up in visits and enquiries.
AI assistants as a channel of their own
Referrals from ChatGPT, Perplexity, Copilot, Gemini and other services are grouped into a channel of their own, with goals and revenue and comparable with search, social and newsletter.
FAQ
Frequently asked questions
Why doesn't Matomo see the AI crawlers on its own?
JavaScript tracking measures browsers that run scripts. Crawlers usually only fetch the HTML and therefore only appear in the server logs. The pipeline reads these logs and transfers the hits to Matomo.
Which crawlers are detected?
The well-known AI crawlers and fetchers, for example from OpenAI, Anthropic, Perplexity, Google, Apple or Common Crawl, plus classic search engines.
Do we need access to the server?
Yes, to the access logs of the web server or an upstream proxy. Where that is not possible, for example with some hosting plans, we look at alternatives.
What about personal data in the logs?
Server logs contain IP addresses. Which data is transferred and how long it is kept, we agree with your data protection team.
How do we measure visitors from ChatGPT and other assistants?
By where the visits come from: referrals from ChatGPT, Perplexity, Copilot, Gemini and other services are grouped into a channel of their own. You see how many visits, goals and revenue come from AI assistants.
Related
Questions about this?
In an intro call we'll clarify where you stand and whether I'm the right fit.
Your contactPeter Boehlke, Matomo consultant from Munich, 17 years in the Matomo team