AI crawlers now show up in review-site server logs more often than most SEO teams realize, and most of that traffic never turns into a citation you can see in Search Console. Some of it is a model reading your pages to learn. Some of it is a model reading your pages to answer someone’s question right now. The difference matters, because it changes what content is worth prioritizing next. Here’s what an AI crawler actually does on a review site, which ones show up most, and what the crawl data says about where to focus a GEO strategy from here.
Quick answer: AI crawlers are bots that AI companies run to fetch and read web pages, either to build training data for a future model or to pull live information into a chat answer. On software review sites, they cluster on product, comparison, and category pages, the same pages buyers ask AI tools about before they build a shortlist.
What an AI crawler actually does when it visits a review site
An AI crawler is a bot an AI company operates to fetch a web page, parse its text, and store or process that content, either as training data for a future model or as live context for answering a specific prompt. Mechanically, it works a lot like a search engine crawler. Where the content ends up is what’s different: a model, not a search index.
In practice, that means the crawler requests a page, reads the rendered or raw HTML depending on the bot, extracts the text (and increasingly the schema markup around it), and may follow internal links to related pages. It doesn’t click buttons, fill out forms, or trigger the ad impressions and session data that show up in Google Analytics, which is a big part of why this traffic is invisible unless you’re reading server logs directly. Before any of that happens, most AI crawlers check robots.txt first. The well-behaved ones, GPTBot, ClaudeBot, and Google-Extended among them, publish their user-agent strings and state that they honor a disallow rule. A handful of newer answer-agents blur that line, since they’re built to act more like a browsing assistant fetching one page on a user’s behalf than a bulk crawler indexing everything it can reach.
Grounding vs. training: two different reasons an AI engine crawls a page
Not every crawl serves the same purpose, and conflating them is where most GEO measurement goes wrong. AI engines pull content from the web for one of two reasons: training or grounding. The buyer demand behind grounding crawls is already mainstream, not a niche edge case: HG Insights data shows nearly one in four US companies (24%) currently carries a high GenAI buying-intent score, meaning they’re actively evaluating or adopting generative AI capability right now. That’s the same population asking AI tools the vendor-comparison questions grounding crawls exist to answer.
Training crawls build the model’s baseline knowledge
Training crawlers, GPTBot, ClaudeBot, Google-Extended, and Applebot-Extended among them, sweep large sections of the web in bulk to build or refresh a model’s foundational dataset. This runs on the model’s schedule, not a buyer’s, tied to a training cycle rather than any single question someone asked. It’s also the crawl type that’s easiest to opt out of: robots.txt is the documented mechanism, and the major labs state that their training crawlers respect it.
Grounding crawls answer a live buyer question
Grounding happens when an AI engine needs to answer a specific prompt right now and fetches a page to ground that answer in current information, rather than relying only on what it learned during training. This is the crawl activity tied directly to a buyer’s live research, and it’s the one worth watching closest. TrustRadius content alone answers roughly 30 million of these LLM grounding calls a year, which is a useful reminder of how much real-time retrieval already runs through review-site content specifically. Training happens on someone else’s timeline. Grounding is the crawl activity a content team can actually move by publishing something better this quarter.
Which AI crawlers show up most on software review sites, and how often
A handful of operators account for nearly all AI crawl traffic on review sites, and the split between them is uneven. OpenAI, Anthropic, and Perplexity dominate the logs, each running both a bulk crawler and a faster-moving answer bot with a different job.
OpenAI: GPTBot and OAI-SearchBot
GPTBot is OpenAI’s training crawler, the one doing the bulk sweeps that feed future model versions. OAI-SearchBot is a separate, faster-moving crawler tied to ChatGPT’s live search feature, fetching pages to answer a user’s question in the moment rather than to train anything. Seeing both in your logs isn’t redundant; it’s two different jobs being done by the same company.
Anthropic: ClaudeBot and Claude-User
ClaudeBot runs the same kind of bulk, training-oriented crawl for Claude. Claude-User is narrower: it fires when someone using Claude explicitly asks the assistant to visit a specific URL, which shows up as a single-page fetch rather than a sweep.
Perplexity: PerplexityBot and Perplexity-User
PerplexityBot indexes continuously to support Perplexity’s answer engine, closer in behavior to a search crawler than a training one. Perplexity-User is the one to watch closely: Perplexity has described it as an agent rather than a bot, which has become a real point of friction with publishers and with Cloudflare over whether it’s obligated to honor robots.txt at all.
Beyond those three, Google-Extended, Applebot-Extended, Amazonbot, and Meta-ExternalAgent make up most of the rest of the field. Each carries a smaller share of review-site crawl volume individually, but they’re still worth a line in your log-review checklist, since a new operator picking up your pages is often the first signal that a new AI engine is starting to notice your category.
Why comparison and “best of” pages get crawled and cited more than single product review pages
Citation tracking across review sites tells a consistent story: activity concentrates on pages that compare or list multiple products, not pages built around a single one. Among review sites, G2 leads on total citation volume, and even there, the citations skew heavily toward comparison and collection pages rather than standalone product review pages. Buyer-side data backs this up: HG Insights currently tracks roughly 63,600 US companies showing high-signal buying intent on customer-reviews topics at any given time, the exact research moment comparison and listicle pages are built to capture.
The reason maps back to how buyers actually use AI tools. Someone asking an AI engine “what is Salesforce” gets answered from general knowledge most of the time. Someone asking “Salesforce vs HubSpot for a 50-person sales team” needs a source that already did that comparison, with specifics the model can lift directly into its answer. A single product review page answers “is this good.” A comparison or listicle page answers “which one, for this situation,” and that’s closer to the actual decision a buyer is trying to make when they reach for an AI tool in the first place.
What crawler analytics data reveals about review-site activity by page type
Crawler analytics tools built for review-site vendors break every crawl down by page type, comparison, product, reviews, full review, pricing, and everything else, and by purpose, grounding or training. That combination is what turns raw log noise into something a content team can act on.
Looked at this way, page type and crawl purpose tend to move together. Comparison pages draw a disproportionate share of grounding crawls specifically, the kind tied to a buyer’s live question rather than a model’s training run. Pricing pages show a similar pattern on a smaller scale. Standalone review pages get crawled too, just less often relative to their volume on the site, which lines up with the citation pattern above rather than contradicting it.
How crawl activity connects to your Citation Share of Voice
Crawl volume and citations aren’t the same metric, and they shouldn’t be read as interchangeable. A page can get crawled constantly and never earn a citation if a competitor’s page answers the question better in the model’s eyes. That’s what makes crawl data a leading indicator rather than a scorecard: it shows where AI attention is concentrating before a single citation shows up, which gives a content team a head start on deciding where to invest next. The full mechanics of how citations get counted and compared against competitors, what TrustRadius calls Citation Share of Voice, are covered in What Is AI Visibility Monitoring? A Buyer’s Guide, worth reading alongside this one rather than repeated here.
See your own product’s crawl and citation activity broken out by page type and AI engine with the GEO Monitoring Dashboard’s Crawler Analytics tab, included in every CVP account at no added cost.
What to check in your own crawler logs before your next content push
Before deciding what to build next, it’s worth confirming a few things directly in your own logs rather than assuming they match the patterns above:
- Which operators are actually visiting your pages
- How that traffic splits between grounding and training
- Which page types draw the most grounding crawls specifically
- Whether your comparison or “vs” pages are getting crawled at a rate that matches their share of citations
If they’re not, that’s a content gap, not a crawler problem.
One thing worth confirming before touching robots.txt: research from Rutgers and Wharton published in December 2025 found that publishers who blocked AI crawlers saw a 23.1% drop in total traffic without reliably reducing how often they got cited. Blocking training crawlers selectively while leaving answer bots open is a reasonable default. Blocking everything on principle tends to cost more than it protects.
Crawl logs tell you where AI attention is going. What determines whether that attention turns into a citation, crawlability requirements, schema, review depth, is a separate question, and it’s the one Why AI Engines Cite Some Vendors and Skip Others answers in full. [Talk to a TrustRadius GEO strategist](link to Contact/Demo Request page) about a citation-worthy content roadmap built on HG Insights market intelligence for your category.
Frequently Asked Questions
What is an AI crawler?
An AI crawler is a bot an AI company runs to fetch and read web pages for use by a large language model, either to build training data for a future model version or to pull current information into a live chat answer. It works similarly to a search engine crawler but feeds a model instead of a search index.
Do AI crawlers respect robots.txt?
Most training-focused crawlers, including GPTBot, ClaudeBot, and Google-Extended, state that they honor robots.txt disallow rules. Some answer-agents are less consistent. Perplexity has described Perplexity-User as an agent rather than a bot, a distinction it has used to argue the rule doesn’t apply the same way, which has caused disputes with publishers and infrastructure providers.
What's the difference between an AI crawler and a search engine crawler?
Both fetch and parse web pages using similar mechanics. A search engine crawler builds a search index that ranks pages for a keyword query. An AI crawler feeds a language model, either during training or in the moment a user asks a question, and the output is a generated answer rather than a ranked list of links.
How often do AI crawlers visit a website?
It varies by operator and by purpose. Training crawlers tend to sweep a site in large batches tied to a model’s training schedule, sometimes weeks or months apart. Grounding and answer crawlers can hit the same page repeatedly within a single day if buyers keep asking questions that page can answer.
Does blocking AI crawlers stop competitors from getting cited instead?
Not reliably. Research from Rutgers and Wharton (December 2025) found that publishers blocking AI crawlers lost 23.1% of total traffic without a consistent drop in how often competitors got cited in their place. Blocking removes your content from consideration; it doesn’t remove the question buyers are asking.
Is crawl volume the same thing as AI visibility?
No. Crawl volume shows how much AI attention a page is getting. Citations and mentions show whether that attention actually turned into your brand showing up in the answer. A page can be crawled often and still lose the citation to a competitor’s page, which is why crawl data and citation data are read together, not as substitutes for each other.
