Thoughtful Things: Building Human-Centric Smart Devices with Small Language Models

TL;DR

This paper introduces a framework for building human-centric smart devices using small, on-device language models that can understand, explain, and act on unconstrained user commands without relying on cloud services.

Contribution

It presents an end-to-end framework combining formal modeling, data synthesis, and generative models to create capable, thoughtful devices that operate entirely on-device.

Findings

Devices can understand and respond to natural language commands.

No labeled data is required for training.

On-device deployment shows practical effectiveness.

Abstract

Everyday devices like light bulbs and kitchen appliances are now embedded with so many features and automated behaviors that they have become complicated to actually use. While such "smart" capabilities can better support users' goals, the task of learning the "ins and outs" of different devices is daunting. Voice assistants aim to solve this problem by providing a natural language interface to devices, yet such assistants cannot understand loosely-constrained commands, they lack the ability to reason about and explain devices' behaviors to users, and they rely on connectivity to intrusive cloud infrastructure. Toward addressing these issues, we propose thoughtful things: devices that leverage lightweight, on-device language models to take actions and explain their behaviors in response to unconstrained user commands. We propose an end-to-end framework that leverages formal modeling, automated training data synthesis, and generative language models to create devices that are both capable and thoughtful in the presence of unconstrained user goals and inquiries. Our framework requires no labeled data and can be deployed on-device, with no cloud dependency. We implement two thoughtful things (a lamp and a thermostat) and deploy them on real hardware, evaluating their practical performance.