ProAgent: Harnessing On-Demand Sensory Contexts for Proactive LLM Agent Systems in the Wild

TL;DR

ProAgent is an end-to-end proactive LLM agent system that uses on-demand sensory contexts and hierarchical perception to provide continuous, in-the-wild assistance with improved accuracy and user satisfaction.

Contribution

It introduces a tiered perception and context extraction framework for continuous sensing and proactive assistance in real-world environments.

Findings

Achieves up to 27.7% higher proactive prediction accuracy.

Reduces false detection by 20.5%.

85% user satisfaction in real-world deployment.

Abstract

Recent studies have begun to explore proactive large language model (LLM) agents that provide unobtrusive assistance by automatically leveraging contextual information, such as in code editing and in-app suggestions. However, most focus on short, task-specific episodes or on-screen contexts, rather than continuously perceiving and assisting users throughout daily life. Enabling such in-the-wild assistance requires continuous sensing of users' surroundings, which can incur substantial system overhead. In this work, we propose ProAgent, an end-to-end proactive agent system that harnesses on-demand sensory contexts to provide in-the-wild assistance. ProAgent first employs on-demand tiered perception to continuously sense users' surroundings by integrating low-cost contextual cues with richer perception on demand, and uses proactive-oriented context extraction to derive hierarchical contexts integrating both sensory contexts and human preferences. ProAgent then employs a context-aware proactive reasoner to infer user needs and invokes external tools to deliver proactive assistance. We implement ProAgent on AR glasses and evaluate it on a public dataset and a real-world dataset. Results demonstrate that ProAgent achieves up to 27.7% higher proactive prediction accuracy and 20.5% lower false detection than state-of-the-art baselines. A user study with 20 participants shows that 85% were satisfied with ProAgent and willing to use it in daily life.