New large-scale traffic analysis suggests website owners are increasingly selective about which AI systems they allow to crawl their content, with AI search and assistant bots gaining ground as training-focused crawlers lose access.
Based on anonymized server logs covering more than 66 billion bot requests across over 5 million websites, Hostinger’s research shows a clear split emerging in how sites treat different categories of AI crawlers.
Training crawlers see sharp decline
The most pronounced shift involves crawlers used to collect data for training large language models. OpenAI’s GPTBot, which is designed to gather content for model improvement, experienced a steep drop in site coverage during the study period, falling from broad access to a small fraction of websites.
Hostinger’s data indicates that this decline is driven primarily by explicit blocking rather than reduced crawling activity. While request volumes from training bots remain substantial, access permissions have tightened significantly across many sites.
This trend reflects growing concern among publishers and site operators about content reuse, intellectual property, and infrastructure costs associated with large-scale AI training crawls.
Search and assistant bots move in the opposite direction
In contrast, bots that retrieve content to answer specific user queries are expanding their reach. OpenAI’s search-focused crawler surpassed 55% average site coverage in Hostinger’s dataset, signaling broader acceptance among site owners.
Other assistant-oriented crawlers also increased access during the same time windows, suggesting that websites are more willing to support AI systems that surface content directly in response to user searches rather than ingest it for future model training.
Because these crawls are typically user-initiated and more targeted, they tend to generate fewer requests per site and are more closely associated with potential discovery benefits.
Traditional search remains steady
Classic search engine crawlers showed relatively little change over the course of the analysis. Google’s primary crawler maintained the highest overall coverage, while other established search bots remained stable.
This consistency highlights the unique position traditional search engines occupy. Blocking these crawlers has immediate visibility consequences, making them less comparable to newer AI-focused bots in site owners’ decision-making.
SEO and marketing crawlers lose ground
Hostinger also observed declining coverage for SEO and marketing-related crawlers. While some tools still maintain a broad footprint, overall access for this category has contracted.
The company attributes this to a combination of more selective crawling strategies by tool providers and increasing use of blocking rules by site operators seeking to limit resource-intensive bot traffic.
A more segmented crawler strategy
The findings point to a more nuanced approach to bot management. Rather than adopting blanket policies, many sites appear to be distinguishing between crawlers that support content discovery and those that extract data without a direct audience-facing benefit.
For organizations evaluating their own policies, the data underscores the importance of aligning crawler access with strategic goals—whether that is maximizing visibility in AI-powered search experiences, limiting server load, or controlling how content is used for model training.
As AI-driven discovery tools continue to evolve, Hostinger’s analysis suggests that crawler access decisions are becoming an increasingly deliberate part of technical and marketing strategy.
