SensorChat: Answering Qualitative and Quantitative Questions during Long-Term Multimodal Sensor Interactions

TL;DR

SensorChat is an innovative end-to-end question-answering system that interprets long-term multimodal sensor data to provide accurate, meaningful responses to both quantitative and qualitative questions in daily life monitoring.

Contribution

It introduces a novel three-stage pipeline leveraging LLMs for interpreting human queries, extracting relevant sensor data, and generating responses, capable of handling long-duration, high-frequency data.

Findings

Achieves 93% higher answer accuracy on quantitative questions compared to state-of-the-art.

Effectively handles both quantitative and qualitative questions in real-time.

Demonstrates capability for real-time cloud and edge deployment.

Abstract

Natural language interaction with sensing systems is crucial for addressing users' personal concerns and providing health-related insights into their daily lives. When a user asks a question, the system automatically analyzes the full history of sensor data, extracts relevant information, and generates an appropriate response. However, existing systems are limited to short-duration (e.g., one minute) or low-frequency (e.g., daily step count) sensor data. In addition, they struggle with quantitative questions that require precise numerical answers. In this work, we introduce SensorChat, the first end-to-end QA system designed for daily life monitoring using long-duration, high-frequency time series data. Given raw sensor signals spanning multiple days and a user-defined natural language question, SensorChat generates semantically meaningful responses that directly address user concerns. SensorChat effectively handles both quantitative questions that require numerical precision and qualitative questions that require high-level reasoning to infer subjective insights. To achieve this, SensorChat uses an innovative three-stage pipeline including question decomposition, sensor data query, and answer assembly. The first and third stages leverage Large Language Models (LLMs) to interpret human queries and generate responses. The intermediate querying stage extracts relevant information from the complete sensor data history. Real-world implementations demonstrate SensorChat's capability for real-time interactions on a cloud server while also being able to run entirely on edge platforms after quantization. Comprehensive QA evaluations show that SensorChat achieves 93% higher answer accuracy than the best performing state-of-the-art systems on quantitative questions. Furthermore, a user study with eight volunteers highlights SensorChat's effectiveness in answering qualitative questions.