LLMSense: Harnessing LLMs for High-level Reasoning Over Spatiotemporal Sensor Traces

TL;DR

LLMSense demonstrates that large language models can effectively perform high-level reasoning over long-term spatiotemporal sensor data, enabling applications like dementia diagnosis and occupancy tracking with high accuracy.

Contribution

This work introduces a novel prompting framework for LLMs to handle high-level reasoning on sensor traces, integrating summarization and selective historical data inclusion for improved performance.

Findings

Achieves over 80% accuracy on dementia diagnosis and occupancy tracking tasks.

Effective framework for high-level reasoning using LLMs on raw and processed sensor data.

Guidelines and insights for leveraging LLMs in sensor-based reasoning applications.

Abstract

Most studies on machine learning in sensing systems focus on low-level perception tasks that process raw sensory data within a short time window. However, many practical applications, such as human routine modeling and occupancy tracking, require high-level reasoning abilities to comprehend concepts and make inferences based on long-term sensor traces. Existing machine learning-based approaches for handling such complex tasks struggle to generalize due to the limited training samples and the high dimensionality of sensor traces, necessitating the integration of human knowledge for designing first-principle models or logic reasoning methods. We pose a fundamental question: Can we harness the reasoning capabilities and world knowledge of Large Language Models (LLMs) to recognize complex events from long-term spatiotemporal sensor traces? To answer this question, we design an effective prompting framework for LLMs on high-level reasoning tasks, which can handle traces from the raw sensor data as well as the low-level perception results. We also design two strategies to enhance performance with long sensor traces, including summarization before reasoning and selective inclusion of historical traces. Our framework can be implemented in an edge-cloud setup, running small LLMs on the edge for data summarization and performing high-level reasoning on the cloud for privacy preservation. The results show that LLMSense can achieve over 80\% accuracy on two high-level reasoning tasks such as dementia diagnosis with behavior traces and occupancy tracking with environmental sensor traces. This paper provides a few insights and guidelines for leveraging LLM for high-level reasoning on sensor traces and highlights several directions for future work.