Bridging the gap between natural user expression with complex automation programming in smart homes

TL;DR

This paper introduces AwareAuto, a system leveraging large language models to enable natural, flexible, and context-aware end-user programming for complex smart home automation, achieving high accuracy in understanding user intentions.

Contribution

The paper presents AwareAuto, a novel LLM-based system that standardizes user expression and handles complex automation tasks in smart homes, improving usability and accuracy over traditional methods.

Findings

Achieved 91.7% accuracy in matching user intentions.

Enabled configuration of complex automation with dynamic parameters and multiple conditions.

Incorporated user interaction for system controllability and usability.

Abstract

A long-standing challenge in end-user programming (EUP) is to trade off between natural user expression and the complexity of programming tasks. As large language models (LLMs) are empowered to handle semantic inference and natural language understanding, it remains under-explored how such capabilities can facilitate end-users to configure complex automation more naturally and easily. We propose AwareAuto, an EUP system that standardizes user expression and finishes two-step inference with the LLMs to achieve automation generation. AwareAuto allows contextual, multi-modality, and flexible user expression to configure complex automation tasks (e.g., dynamic parameters, multiple conditional branches, and temporal constraints), which are non-manageable in traditional EUP solutions. By studying realistic, complex rules data, AwareAuto gains 91.7% accuracy in matching user intentions and feasibility. We introduced user interaction to ensure system controllability and usability. We discuss the opportunities and challenges of incorporating LLMs in end-user programming techniques and grounding complex smart home contexts.