Thou Shalt Not Prompt: Zero-Shot Human Activity Recognition in Smart Homes via Language Modeling of Sensor Data & Activities

TL;DR

This paper introduces a novel zero-shot human activity recognition method for smart homes that models sensor data and activities using natural language embeddings, avoiding reliance on prompt-based large language models.

Contribution

The paper proposes a new approach that uses language modeling of sensor data and activities for zero-shot HAR, eliminating the need for prompt-based LLM classification.

Findings

Effective zero-shot recognition across six datasets

Outperforms prompt-based LLM methods in accuracy and privacy

Demonstrates robustness without external LLM prompting

Abstract

Developing zero-shot human activity recognition (HAR) methods is a critical direction in smart home research -- considering its impact on making HAR systems work across smart homes having diverse sensing modalities, layouts, and activities of interest. The state-of-the-art solutions along this direction are based on generating natural language descriptions of the sensor data and feeding it via a carefully crafted prompt to the LLM to perform classification. Despite their performance guarantees, such ``prompt-the-LLM'' approaches carry several risks, including privacy invasion, reliance on an external service, and inconsistent predictions due to version changes, making a case for alternative zero-shot HAR methods that do not require prompting the LLMs. In this paper, we propose one such solution that models sensor data and activities using natural language, leveraging its embeddings to perform zero-shot classification and thereby bypassing the need to prompt the LLMs for activity predictions. The impact of our work lies in presenting a detailed case study on six datasets, highlighting how language modeling can bolster HAR systems in zero-shot recognition.