Google's Search team and its Chrome developer tools team are now giving site owners conflicting technical instructions on llms.txt, a machine-readable file format that has divided the SEO industry for months. On May 15, 2026, Google published a new guide through the Google Search Central Blog that explicitly tells site owners to ignore the file. Days earlier, Google's Lighthouse tool shipped version 13.3, which added a new Agentic Browsing category, and with it, an llms.txt audit that checks whether a site provides the file and flags server errors when retrieving it.
Google's Search Central Guidance: Standard SEO Still Applies
The new Google’s Search Central guide covers the importance of providing valuable, unique, non-commodity content; tips about providing local, shopping, image, and video content; mythbusting common AEO and GEO misconceptions; and initial guidance related to AI agents.
Google opens the guide by confirming that foundational SEO best practices remain relevant for generative AI search, stating its AI features are "rooted in our core Search ranking and quality systems" and rely on retrieval-augmented generation (RAG) and query fan-out to surface content from the Search index. The guide states plainly that there is no separate optimization layer for AI search. From Google Search's perspective, optimizing for generative AI search is optimizing for the search experience, and thus still SEO.
On the technical side, the guidance is similarly familiar. Pages must be indexed and eligible for snippets to appear in generative AI features. To maximize a site's visibility in generative AI search features, Google states that content must be crawlable, as its generative AI models use publicly accessible, crawlable content to learn patterns and provide relevant, grounded responses.
The guide also addresses what Google calls non-commodity content, a distinction it frames as the single most consequential factor for AI visibility. Google contrasts commodity content, such as "7 Tips for First-Time Homebuyers," with non-commodity content, such as "Why We Waived the Inspection & Saved Money: A Look Inside the Sewer Line," where the distinction is whether content provides unique insight beyond common knowledge.
What Google Says Site Owners Can Skip
The guide includes a "Mythbusting generative AI search" section listing tactics it calls unnecessary for Google Search. Google explicitly lists tactics it says site owners should not pursue: llms.txt files, content "chunking," AI-specific rewriting, pursuing inauthentic brand mentions, and over-indexing on structured data for AI purposes.
On llms.txt specifically, the May 15 Search Central guide states that site owners do not need to create machine-readable files, AI text files, markup, or Markdown to appear in generative AI search, and notes that Google may crawl such files without treating them in any special way. On the question of whether creating pages targeting every possible AI sub-query is a viable strategy, Google states that doing so primarily to manipulate rankings or generative AI responses violates its scaled content abuse spam policy.
On structured data, the guide says it is not required for generative AI search, and there is no special schema.org markup to add, though Google recommends continuing to use it as part of an overall SEO strategy for rich results eligibility.
What Lighthouse 13.3 Now Checks
Lighthouse 13.3.0, released May 7, 2026, moved the Agentic Browsing category from experimental into the default configuration. The Agentic Browsing category evaluates how well a site is constructed for machine interaction through a set of deterministic audits; unlike other Lighthouse categories, it does not produce a weighted average score from 0 to 100, because the standards for the agentic web are still emerging and the current focus is to gather data and provide actionable signals rather than a definitive ranking.
The audits are part of Chrome's emerging "Agentic Browsing" category, which evaluates whether sites are structured for machine interaction. The checks include WebMCP integration, agent accessibility, layout stability, and llms.txt. Agents rely on the accessibility tree as their primary data model, and Lighthouse filters a specific subset of accessibility audits considered critical for machine interaction, such as ensuring every interactive element has a programmatic name.
On llms.txt, Google's Chrome for Developers documentation states that without this file, agents may spend more time crawling the site to understand its high-level structure and primary content. Lighthouse flags pages if a server error occurs when attempting to retrieve the file. If the file is not provided and returns a 404, the audit is marked as Not Applicable, as providing the file is optional at the moment.
The Lighthouse documentation describes llms.txt as a way to provide "a machine-readable summary of a website's content, specifically designed for LLMs and AI agents."
Two Google Products, Two Positions
The Lighthouse documentation does not directly conflict with Google's advice on optimizing for generative AI features because these audits focus on AI agents and browser tools, not Google Search rankings. However, the distinction is a product-team distinction, not a widely communicated one, and guidance is split between different Google developer sites, which can lead to conflicting instructions when comparing Lighthouse or its llms.txt documentation with Google's Search docs.
Google's Search Advocate John Mueller addressed the apparent contradiction on Bluesky on May 20, 2026, responding to a question from SEO consultant Lily Ray about why Google's own developer properties publish llms.txt files and Markdown pages despite advising against them. Mueller explained that AI coding systems can be efficient and accurate if they can easily read and parse developer documentation, so it can help to give them a way to understand context and a simplified version of the reference page in Markdown, while noting this is more of a temporary crutch, perhaps to save some tokens, since systems can read HTML just fine. He added that for non-developer sites, he does not think this makes much sense, even with more agentic traffic in the future, noting that most sites are not currently receiving a lot of that traffic.
Mueller's position on llms.txt for Google Search itself has been consistent across previous statements. He explicitly compared the file to the keywords meta tag, a tag search engines have been ignoring for over a decade, and Gary Illyes confirmed at Google Search Central Live that Google does not support llms.txt and has no plans to do so.
The Agentic Browsing Category in Context
The Agentic Browsing category is separate from SEO audits and indicates that llms.txt helps browser-based agents understand site structure, not improve search rankings or AI citations. The audits align with a broader direction flagged by Google Cloud AI engineering director Addy Osmani, who outlined "Agentic Engine Optimization" considerations in April, including cleaner semantic structure, token-efficient content, and llms.txt discovery layers.
It still matters for users because agents are supposed to reliably fill out forms, make bookings, or compare products, provided pages are built to be machine-readable. The Lighthouse documentation frames those browser-agent interactions as distinct from the Search ranking systems that power AI Overviews and AI Mode.
Google has not commented on the documentation gap between the two product teams.
Practical Implications for SEO and Technical Teams
For site owners and SEO practitioners, the immediate practical separation is this: llms.txt is not a factor in Google Search rankings, AI Overviews, or AI Mode visibility, based on the May 15 Search Central guide. However, the presence of the file, or a server error when it is requested, is now a named signal in Lighthouse's default Agentic Browsing audit. Sites with server misconfigurations that generate errors on unfamiliar file requests may surface a flag in that category. For developer-documentation sites or large technical properties, the file may offer efficiency benefits for AI coding agents, as Mueller outlined. For most consumer-facing sites, the Search Central guide's foundational advice, crawlable pages, indexed content, non-commodity writing, and strong technical hygiene, remains the primary optimization target for AI search visibility.
Creating a basic llms.txt file is simple for many sites, but maintaining it raises questions, given that Google Search states it is unnecessary for AI Search visibility.
