GLOSS: Group of LLMs for Open-Ended Sensemaking of Passive Sensing Data for Health and Wellbeing

TL;DR

GLOSS is a novel system that leverages multiple large language models to interpret passive sensing data for health and wellbeing, enabling open-ended, complex insights that outperform existing retrieval-based methods.

Contribution

The paper introduces GLOSS, a new multi-LLM system capable of open-ended sensemaking and complex data triangulation, advancing passive sensing data analysis for health applications.

Findings

GLOSS achieves 87.93% accuracy in sensemaking tasks.

GLOSS demonstrates 66.19% consistency in insights.

Outperforms RAG with significantly higher accuracy and consistency.

Abstract

The ubiquitous presence of smartphones and wearables has enabled researchers to build prediction and detection models for various health and behavior outcomes using passive sensing data from these devices. Achieving a high-level, holistic understanding of an individual's behavior and context, however, remains a significant challenge. Due to the nature of passive sensing data, sensemaking -- the process of interpreting and extracting insights -- requires both domain knowledge and technical expertise, creating barriers for different stakeholders. Existing systems designed to support sensemaking are either not open-ended or cannot perform complex data triangulation. In this paper, we present a novel sensemaking system, Group of LLMs for Open-ended Sensemaking (GLOSS), capable of open-ended sensemaking and performing complex multimodal triangulation to derive insights. We demonstrate that GLOSS significantly outperforms the commonly used Retrieval-Augmented Generation (RAG) technique, achieving 87.93% accuracy and 66.19% consistency, compared to RAG's 29.31% accuracy and 52.85% consistency. Furthermore, we showcase the promise of GLOSS through four use cases inspired by prior and ongoing work in the UbiComp and HCI communities. Finally, we discuss the potential of GLOSS, its broader implications, and the limitations of our work.