SmartBench: Evaluating LLMs in Smart Homes with Anomalous Device States and Behavioral Contexts

TL;DR

SmartBench introduces a novel dataset to evaluate large language models' ability to detect anomalies in smart home environments, revealing current models' limitations in accurately identifying abnormal device states and transitions.

Contribution

This paper presents the first smart home dataset designed for LLMs, focusing on normal and anomalous device states and transitions, and evaluates 13 models' anomaly detection performance.

Findings

Most LLMs perform poorly on anomaly detection tasks.

Claude-Sonnet-4.5 achieves 66.1% accuracy on context-independent anomalies.

Models struggle with context-dependent anomaly detection.

Abstract

Due to the strong context-awareness capabilities demonstrated by large language models (LLMs), recent research has begun exploring their integration into smart home assistants to help users manage and adjust their living environments. While LLMs have been shown to effectively understand user needs and provide appropriate responses, most existing studies primarily focus on interpreting and executing user behaviors or instructions. However, a critical function of smart home assistants is the ability to detect when the home environment is in an anomalous state. This involves two key requirements: the LLM must accurately determine whether an anomalous condition is present, and provide either a clear explanation or actionable suggestions. To enhance the anomaly detection capabilities of next-generation LLM-based smart home assistants, we introduce SmartBench, which is the first smart home dataset designed for LLMs, containing both normal and anomalous device states as well as normal and anomalous device state transition contexts. We evaluate 13 mainstream LLMs on this benchmark. The experimental results show that most state-of-the-art models cannot achieve good anomaly detection performance. For example, Claude-Sonnet-4.5 achieves only 66.1% detection accuracy on context-independent anomaly categories, and performs even worse on context-dependent anomalies, with an accuracy of only 57.8%. More experimental results suggest that next-generation LLM-based smart home assistants are still far from being able to effectively detect and handle anomalous conditions in the smart home environment. Our dataset is publicly available at https://github.com/horizonsinzqs/SmartBench.