Website owners are now receiving roughly three times more automated crawl requests from OpenAI than they were before August 2025. On April 23, 2026, Botify and Nectiv published an analysis of more than 7 billion OpenAI log file events, finding that OpenAI's web crawl has tripled since August 2025.
The research was conducted by Chris Long, co-founder of SEO and AI-engine optimization consultancy Nectiv, in partnership with Botify, the enterprise SEO and crawl intelligence platform. The dataset spans November 2024 through March 2026 and draws on Botify's log file infrastructure, which the company processes on behalf of large enterprise clients across retail and e-commerce, media and publishing, healthcare, software, travel, and marketplaces.
OpenAI's Three Crawlers and What Each One Does
To understand the data, it is necessary to understand that OpenAI operates three distinct crawlers, each with a different function. OpenAI uses web crawlers and user agents to perform actions for its products, either automatically or triggered by a user request. The company uses OAI-SearchBot and GPTBot robots.txt tags to enable webmasters to manage how their sites and content work with AI, with each setting operating independently. A webmaster can allow OAI-SearchBot to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training OpenAI's generative AI foundation models.
ChatGPT-User represents a user-initiated action. When someone instructs ChatGPT to fetch or interact with a page directly, this is the agent that executes the request. GPTBot is OpenAI's general-purpose training crawler, designed to collect data that can be used to improve the foundational knowledge of its language models. OAI-SearchBot is the web search crawler: it operates when ChatGPT performs a search that is not directly triggered by a user, sourcing results for responses that involve real-time web retrieval.
If a site has allowed both bots, OpenAI may use the results from just one crawl for both use cases to avoid duplicative crawling.
OAI-SearchBot and GPTBot Both Surged After GPT-5
OpenAI crawl activity roughly tripled after GPT-5 launched in August 2025, and OAI-SearchBot now generates more log events than GPTBot, pointing to more activity tied to ChatGPT search.
OAI-SearchBot, which retrieves content when ChatGPT performs web searches, recorded about 3.5x more events after August 2025, working out to roughly 2.2 billion additional events in Botify's dataset. GPTBot, which collects training data, recorded about 2.9x more events over the same period, another 1.8 billion events.
Before GPT-5 launched, the two automated bots ran at comparable volumes. OAI-SearchBot and GPTBot ran at roughly even volumes in Botify's dataset before GPT-5, with a ratio of about 0.95 search events per training event. After GPT-5, that ratio rose to about 1.14. The shift means OpenAI is now spending more crawl resources on live web search than on training data collection, a reversal from the pattern that existed less than a year ago.
ChatGPT-User Events Dropped 28% in the Same Period
While the automated crawlers surged, the third user agent moved in the opposite direction. ChatGPT-User moved in the opposite direction, with Long reporting a 28% drop in ChatGPT-User log events between December 2025 and March 2026. ChatGPT-User fires when a ChatGPT session fetches a page on behalf of a user, so the drop measures logged user-initiated fetches rather than ChatGPT usage overall.
The analysis offers two possible explanations for the decline, without settling on either. One possibility is that fewer sessions may be triggering real-time page fetches. The other, suggested by Botify's team, is that OpenAI may be relying more on stored or indexed resources, reducing the need to fetch pages in real time. This is similar to how Gemini relies on Google's index instead of crawling pages on demand when grounding responses, and may be an indication that OpenAI's HTML web index is becoming even more comprehensive, rather than a signal of a drop in ChatGPT users.
Supporting the user-decline interpretation, SimilarWeb found that ChatGPT dropped from 86.7% traffic share in January 2025 to just 64.5% by January 2026, and Sistrix found that ChatGPT usage plateaued around late 2025 and showed usage also dropping.
Search Bot Increases Varied Sharply by Industry
The post-GPT-5 search bot increases varied by industry. Healthcare sites saw about 740% more OAI-SearchBot activity after launch; Media and Publishing, 702%; and Marketplaces, Software, and Retail, 190–216%. Travel sites had the smallest rise at 30%.
The balance between search and training crawling also differs by sector. Long reports a +256% OAI-SearchBot to GPTBot crawl difference for Media/Publishing, the largest gap. Software and Internet sites lean toward search, while Healthcare and Retail favour training, at -50% and -33% respectively, with GPTBot more active overall on those sites.
Botify and Long suggest OpenAI routes prompt types differently: news inquiries trigger live search, while health and product queries rely on trained knowledge.
OpenAI's Crawl Volume Still Dwarfed by Google, but the Gap Is Closing
Despite the tripling in activity, OpenAI's crawl footprint remains a fraction of Google's. In the last month covered by the dataset, Googlebot, both desktop and smartphone, registered 18.2 billion events, while OpenAI's combined crawlers generated 887 million. That places OpenAI at roughly 4% of Google's crawl volume. Bing, for context, registered 5.49 billion events in the same period, meaning OpenAI represents about 14% of Bing's crawl.
The year-over-year trajectory shows how quickly that gap has narrowed. In the equivalent 30-day window in 2025, Google crawlers registered 15 billion events while OpenAI registered 207 million, placing ChatGPT at just 1.38% of Google's total crawl. In one year, that share moved from 1.38% to 4%.
What This Means for Crawl Budget and robots.txt Management
The practical consequence for SEOs and site owners is that AI crawlers now represent a meaningful and growing portion of total server load. Because GA4 is blind to AI crawler visits, as AI fetchers don't trigger client-side JavaScript analytics and produce no sessions, no events, and no attribution in GA4, marketers may see flat traffic figures while AI engines generate significant crawl activity in the background.
That invisibility has direct crawl budget implications. The Botify and Nectiv data show OAI-SearchBot is now the dominant OpenAI crawler, yet it is the one most commonly overlooked in robots.txt configurations. OAI-SearchBot now generates more log events than GPTBot, and sites that block only GPTBot are not blocking the bot OpenAI says is used to surface websites in ChatGPT search answers. Sites that block OAI-SearchBot may be excluding themselves from ChatGPT search answers.
For sites that want to control which OpenAI systems access their content, the two bots must be managed separately and deliberately. OpenAI uses OAI-SearchBot and GPTBot robots.txt tags to enable webmasters to manage how their sites and content work with AI. With each setting independent of the other, a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training OpenAI's generative AI foundation models. Reviewing server logs directly, rather than relying on analytics platforms, remains the only reliable method for measuring which OpenAI bots are active on a given site, at what frequency, and whether existing robots.txt directives are being respected.
Dataset Scope and Commercial Disclosure
The dataset covers Botify's enterprise clients in retail, e-commerce, technology, publishing, travel, and marketplaces. The analysis was conducted by Long as a guest author on Botify's blog. Botify sells log file analysis and AI bot management software, and the post promotes a follow-up webinar and a product demo. The dataset skews toward large enterprise websites rather than a representative cross-section of the web.
Botify's findings line up with patterns other vendors have reported. An Alli AI analysis found OpenAI's ChatGPT-User made 3.6x more requests than Googlebot in a smaller WordPress-heavy sample. A Hostinger analysis found OAI-SearchBot's website coverage reaching 55% while GPTBot coverage fell. Akamai's recent bot traffic report showed OpenAI leading AI bot traffic to publishing sites.
