SAGE: Smart home Agent with Grounded Execution
Dmitriy Rivkin, Francois Hogan, Amal Feriani, Abhisek Konar, Adam, Sigal, Steve Liu, Greg Dudek

TL;DR
SAGE is a novel smart home agent that leverages grounded execution and dynamic prompt trees to enhance LLM capabilities, enabling flexible, scalable, and context-aware interactions with smart devices, achieving a 75% success rate on new tasks.
Contribution
This paper introduces SAGE, a new framework that combines LLMs with grounded execution and dynamic prompting to improve smart home automation and user interaction.
Findings
SAGE achieves a 75% success rate on 50 new smart home tasks.
Outperforms existing LLM-based baselines with a 30% success rate.
Supports flexible device control and user preference management.
Abstract
The common sense reasoning abilities and vast general knowledge of Large Language Models (LLMs) make them a natural fit for interpreting user requests in a Smart Home assistant context. LLMs, however, lack specific knowledge about the user and their home limit their potential impact. SAGE (Smart Home Agent with Grounded Execution), overcomes these and other limitations by using a scheme in which a user request triggers an LLM-controlled sequence of discrete actions. These actions can be used to retrieve information, interact with the user, or manipulate device states. SAGE controls this process through a dynamically constructed tree of LLM prompts, which help it decide which action to take next, whether an action was successful, and when to terminate the process. The SAGE action set augments an LLM's capabilities to support some of the most critical requirements for a Smart Home…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · AI in Service Interactions · Speech and dialogue systems
MethodsSparse Evolutionary Training
