Understanding the new era where AI bots, not humans, consume your content first.
“No, this can’t be right,” I said to myself. “According to this, they’re getting over 60% of their entire site traffic from AI bots.”
But when I checked the new custom analytics system we were building for a client a few months back, the figures held up. Unnoticed by most of us, a quiet revolution had been underway.
Something significant has happened to web traffic, and most business owners haven’t noticed. As of 2025, bot traffic now accounts for 51% of all internet activity. More importantly, a growing slice of that comes from AI systems like ChatGPT, Claude, and Perplexity. Not malicious scrapers, but the engines that increasingly answer your customers’ questions.
This matters because the way people find information is fundamentally changing. For over two decades, search engines sent people to websites. Now, AI tools read websites and deliver answers directly. Fewer clicks, but the clicks that do come through tend to be from more engaged, better-informed visitors.
The new visibility problem
The real risk isn’t declining traffic, it’s invisibility. If AI systems can’t access or understand your website, you simply won’t appear when someone asks “Who provides X service in my area?” or “What’s the best solution for Y problem?”
This creates a strategic choice. Block AI crawlers to protect your content, and you disappear from the primary way people will discover businesses. Allow them, and your expertise feeds the systems that guide buying decisions.
For most commercial businesses, invisibility is the greater threat.
Training bots vs search bots
Not all AI crawlers are equal. Training bots (like GPTBot and ClaudeBot) collect content to build the AI’s long-term knowledge. Search bots (like OAI-SearchBot and PerplexityBot) fetch information in real-time to answer user questions, and crucially, they can cite you and send traffic your way.
The smart approach is selective. Allow search bots (they drive traffic), and make a conscious decision about training bots based on whether you’re comfortable with your content informing AI models.
From keywords to entities
Traditional SEO focused on keywords, matching the phrases people type. AI systems think differently. They understand entities, i.e. distinct concepts with relationships to other concepts.
When someone asks an AI about “running trainers,” it doesn’t just pattern-match text. It understands the concept connects to marathon training, arch support, specific brands, and biomechanics. Content that covers these relationships comprehensively gets selected; thin, keyword-stuffed pages get ignored.
What makes AI cite you
Two factors determine whether AI systems reference your content:
Authority through consensus
AI models are designed to minimise hallucinations by seeking agreement across sources. Your content should align with established facts while offering genuine expertise.
Original information
This is the real currency. An AI cannot predict the results of research you conducted, data you collected, or insights from your direct experience. It must cite you to provide that information. Publishing original perspectives and data is the most effective way to earn citations.
Structuring content for AI extraction
AI systems favour content that’s easy to process. This means:
Put answers first
The traditional approach of building to a conclusion doesn’t work here. State your key point immediately, then elaborate.
Use clear hierarchy
Headings aren’t just styling, they define relationships between ideas. Question-based subheadings followed by direct answers create extractable Q&A pairs.
Structure for scanning
Lists, tables, and summary sections are easier for AI to parse than dense paragraphs. You’re essentially pre-formatting your content for the AI’s output.
The measurement challenge
There’s no “Search Console” for ChatGPT. Measuring success requires new approaches: tracking whether your brand appears in AI responses to relevant questions, monitoring the quality (not just quantity) of referral traffic from AI sources, and watching for correlations between AI crawl activity and increases in branded searches.
Early data suggests visitors from AI sources spend more time on sites and convert better — they’ve already been filtered and informed by the AI before arriving.
What’s coming next
By 2028, AI agents won’t just answer questions, they’ll complete transactions. Book travel, compare suppliers, make purchases. The interface disappears entirely.
The businesses that thrive will be those that are easiest for machines to understand and interact with. Structured data, clear entity relationships, and machine-readable information about products and services become competitive advantages.
The practical takeaway
If you’re running a business website, here are concrete changes worth considering:
Review your robots.txt file
Check whether you’re inadvertently blocking AI search bots. At minimum, ensure OAI-SearchBot (ChatGPT Search) and PerplexityBot can access your site. These are the crawlers most likely to cite you and send traffic. You can block training bots like GPTBot separately if you’re uncomfortable with your content being used for model training.
Check your server logs, not just Analytics
Most AI crawlers won’t appear in Google Analytics because they don’t execute JavaScript, they fetch your HTML content but ignore the scripts, so the tracking code never fires. Server logs capture every request regardless, showing which AI systems are crawling your site, how often, and which pages interest them. High crawl activity with little AI referral traffic suggests your content is being read but not cited — typically a content issue rather than a technical one.
Fix errors that bots encounter
If AI crawlers hit 404 pages or server errors, they reduce crawl frequency and your information becomes stale in their systems. Run your logs through a filter for bot user-agents and address any broken paths they’re requesting.
Restructure key pages around questions and answers
Take your most important service or product pages and add clear subheadings phrased as questions your customers actually ask, followed immediately by direct answers. This creates extractable Q&A pairs that AI systems can lift into responses.
Add summary sections to longer content
A “key points” box at the top of detailed pages essentially writes the AI’s summary for it, giving you more control over how your expertise gets represented.
Implement structured data properly
Beyond basic schema, use the sameAs property to link your organisation to authoritative profiles (LinkedIn company page, industry directories, Wikidata if you’re notable enough). This helps AI systems verify that your website belongs to a real, established entity.
Publish what only you know
Original data, survey results, case studies with specific outcomes, expert perspectives from your actual experience. AI systems must cite sources for information they can’t generate themselves. This is your most reliable path to appearing in AI-generated answers.