Blog
AI crawlers distort your analytics: bots vs humans
In June 2026, automated traffic passed human traffic on the web for the first time: Cloudflare measured 57 percent of requests to HTML pages coming from bots. A large share of that is AI crawling, models reading your pages to answer prompts, build search indexes, or train. To your analytics script, a crawler looks like a very fast visitor, and it quietly inflates your numbers. Here is how to spot bot traffic, what it does to your metrics, and how to measure your real visitors again.
MetriXs editorial team · Published
The web tipped over: bots now outnumber people
For years, the rule of thumb was that roughly a third of web traffic was automated. That era is over. In June 2026, Cloudflare reported that automated requests had crossed 50 percent of all web traffic for the first time, reaching about 57 percent of requests to HTML pages. Matthew Prince, the company’s CEO, noted that his own forecast had placed that crossover in 2027. It arrived more than half a year early.
The main driver is artificial intelligence. Every time an AI assistant cites your website in an answer, a crawler fetched that page shortly before. Models crawl to answer user prompts, to build AI search indexes, and to train on new content. Research by Fastly already put automated crawling at around 80 percent of AI-related bot traffic in 2025, and the volume has kept climbing since.
Fake crawlers are not a fringe problem
So if you read your raw logs and see “GPTBot visited my site 500 times”, a large part of that may not be OpenAI at all. Scrapers, SEO tools, and impersonators borrow the names of respectable crawlers precisely because many site owners treat those names as trustworthy. Counting crawler names at face value means counting traffic that never happened, or attributing it to the wrong company.
What bot traffic does to your metrics
Once bots are inside your reports, the damage spreads to almost every number you use for decisions:
- Visitors and pageviews drift upward. Crawler volume fluctuates with model release cycles, not with your marketing. A spike in traffic may be an AI model re-crawling your docs, not a campaign that worked.
- Bounce rate loses its meaning. A crawler fetches one page and leaves. It is a textbook bounce. As bot share grows, your bounce rate grows with it, even if human behavior never changed.
- Conversion rate drifts downward. The denominator (visitors) inflates while the numerator (conversions) stays human-only. Your conversion rate drops without anything actually getting worse.
- Session duration collapses toward zero for bot sessions, dragging your averages down.
- AI referral numbers become unusable. Real clicks from ChatGPT or Perplexity answers are a small, valuable signal. Crawler visits are hundreds of times more frequent. If both end up in the same bucket, the signal drowns in the noise.
For agencies and teams reporting to clients or management, this is quietly corrosive: two reports of the same site, a week apart, can tell completely different stories purely because of crawler activity.
How to separate bots from real visitors
Fully separating the two requires looking at signals a casual faker cannot easily reproduce:
- Verify the claim, not the name. Serious crawlers publish their IP ranges, and a reverse DNS check can confirm that a “Googlebot” really came from Google. Anything that fails verification should not be counted as that crawler.
- Watch behavior, not just headers. Real visitors execute JavaScript, move the mouse, scroll, and load assets over multiple requests. Bare HTTP clients that fetch one page and vanish within a second behave differently in ways that can be measured.
- Compare browser data with server logs. A browser-based script only fires when JavaScript runs; server logs see every request. The gap between the two is a rough map of your automated traffic.
Doing all of this by hand for every report is not realistic. This is where MetriXs does the work for you: every incoming event is checked automatically, and known bots and crawlers are filtered before they ever reach your reports. On Team and Pro plans, the Security tab includes a blocked-bots feed, so you can see exactly what was filtered instead of wondering whether your numbers are clean. The filtering happens on our servers, before storage, so your dashboard shows people, not scripts.
Keep the AI signal you actually want
None of this means AI traffic should be thrown away. Two completely different things hide under the label “AI traffic”:
- Crawlers. Machines reading your content. Useful for visibility in AI answers, but they are not visitors and do not belong in your visitor metrics.
- Referrals. Real people who read an AI answer, clicked the link to your site, and landed on your page. These are genuine visitors, often well-informed and close to a decision.
MetriXs keeps the two apart. Crawlers are filtered out of your core metrics, and real arrivals from ChatGPT, Perplexity, Gemini, Copilot, and other assistants are surfaced in the dedicated AI traffic panel, available on every plan, with bounce rate, session duration, and revenue for commerce sites. That gives you a clean answer to the question that actually matters: are people finding me through AI, and do they engage? How to grow that visibility is covered in our guides on Generative Engine Optimization and measuring AI referrals.
Conclusion
The crossover has happened: bots now generate the majority of web traffic, and a growing share of it is AI crawling. Analytics that count crawlers as visitors will keep getting noisier, and decisions made on those numbers will keep getting shakier. The fix is not to abandon analytics but to filter the machines out and keep the one AI signal worth having: real people arriving from AI answers. MetriXs does the filtering automatically, shows you what it blocked, and gives you the AI referral data separately. You can test all of it on the free plan.
Frequently asked questions
- How much of my website traffic is bots?
- It varies a lot by site and page type. Web-wide, Cloudflare measured about 57 percent of requests to HTML pages as automated in June 2026, and content-heavy sites attract more crawling than average. Signs in your own data: sessions with a single pageview and near-zero duration, traffic spikes you cannot connect to any campaign, and traffic that never appears in your browser-based reports.
- Do AI crawlers hurt my search rankings?
- Not directly. AI crawlers are separate from Googlebot, and being crawled by them is not a ranking penalty. The real costs are practical: wasted server resources, skewed analytics, and the fact that some crawlers do not respect robots.txt. Blocking all AI crawlers also removes your path to being cited in AI answers, which is becoming a real traffic channel.
- Does MetriXs count AI crawlers as visitors?
- No. Every incoming event is checked automatically and known bots and crawlers are filtered before storage, so they never reach your reports or your plan limits. Real people arriving from ChatGPT, Perplexity, Gemini, Copilot and other assistants are counted as visitors and shown separately in the AI traffic panel.
- Why are AI crawlers visiting my site at all?
- For three reasons: to fetch pages they are about to cite in an answer to a user prompt, to build the indexes behind AI search features, and to collect training material. More AI usage means more crawling, which is why bot traffic keeps growing. Platforms such as Cloudflare have started blocking some crawler classes by default because publishers pushed back on uncontrolled scraping.
- Should I block AI crawlers in robots.txt?
- That depends on your goals. If appearing in AI answers matters to you, blocking the answer and search crawlers works against you. The better combination is to allow the crawlers you want citing you, and filter all crawler traffic out of your analytics so it never pollutes your visitor metrics. That is exactly how MetriXs handles it.
Measure people, not bots
MetriXs filters known bots and crawlers automatically, tracks real AI referrals on every plan, and hosts everything in the EU. Free plan, paid tiers from 6 €/month.
Try it free →Related: Generative Engine Optimization · Is ChatGPT recommending your business? · Website security headers checklist · Cookieless tracking: how it works · Analytics MCP server