Google has published new research outlining an approach to extracting user intent directly from on-device interactions, offering a glimpse into how the company is thinking about the next generation of privacy-preserving AI assistants.
The work focuses on understanding what a user is trying to accomplish by observing their interactions with mobile apps or web interfaces, without transmitting raw behavioral data back to centralized servers. According to the researchers, the method not only protects user privacy but can outperform large multimodal models running in data centers.
Small models, local processing
Rather than relying on a single, large multimodal system, the research proposes decomposing intent extraction into two distinct tasks handled by smaller models operating locally on a device or browser.
In the first stage, an on-device model summarizes each individual user interaction. These interactions are treated as a sequence—what the researchers refer to as a trajectory—representing a user’s journey through an application or website.
In the second stage, a separate model processes the sequence of interaction summaries to infer the user’s overall intent. By splitting the problem this way, the system avoids the computational and privacy costs associated with sending screenshots, text, or behavioral logs off-device.
The researchers report that this two-stage design consistently outperformed both smaller baseline models and a state-of-the-art multimodal large language model across multiple datasets.
Inferring intent from interface actions
The approach builds on earlier research into extracting intent from screenshots and action descriptions, but adapts it for constrained, on-device environments. Each step in a user’s trajectory is represented by two observable components: the visual state of the interface and the action the user performs, such as tapping, typing, or navigating.
From these signals, the system attempts to generate intent descriptions that meet three criteria: they must accurately reflect what occurred, include all information necessary to reproduce the sequence, and exclude irrelevant detail.
This is a nontrivial problem. The same visible actions can reflect different underlying motivations, and even human annotators often disagree when labeling intent. Prior studies cited in the paper show that human agreement on intent extraction rarely exceeds 80%.
A deliberate two-stage design
After testing alternative strategies, including chain-of-thought style reasoning, the researchers settled on a two-stage approach that mimics structured reasoning without requiring complex inference from a single small model.
In the first stage, each interaction is summarized using prompt-based techniques. These summaries include a description of the screen and the user’s action. The model is also allowed to speculate about intent, but that speculative component is explicitly discarded—a counterintuitive step that the researchers found improves overall output quality.
In the second stage, a model fine-tuned on curated training data generates a single intent description from the sequence of summaries. During training, the researchers had to address a tendency for the model to hallucinate missing details. They mitigated this by refining the target intent labels to include only information supported by the summaries themselves.
Privacy, ethics, and limitations
A central theme of the research is privacy preservation. Because processing occurs on-device, sensitive interaction data does not need to be transmitted to Google’s servers. This design choice aligns with broader industry efforts to balance personalization with user trust.
The authors also acknowledge potential risks. An autonomous agent capable of inferring intent could take actions misaligned with a user’s goals if guardrails are not carefully designed. As a result, the paper emphasizes the need for strong constraints and transparency in any real-world deployment.
There are also practical limitations. The experiments were conducted only on Android and web environments, using English-language data from U.S.-based users. The results may not generalize to other platforms, languages, or cultural contexts.
What this signals about Google’s direction
The research does not indicate that this intent extraction system is currently deployed in search or consumer products. Instead, it frames the work as foundational.
The paper highlights two potential applications: proactive assistance, where an on-device agent helps users complete tasks more efficiently, and personalized memory, where devices retain structured knowledge of past activities.
Taken together, the findings suggest a future where smaller, specialized models embedded in devices observe user interactions and intervene selectively based on inferred goals. While still experimental, the work points toward a more agent-driven, privacy-conscious AI ecosystem—one that operates closer to the user rather than in distant data centers.
