SensorLM: Learning the Language of Wearable Sensors

TL;DR

SensorLM introduces a new foundation model for wearable sensor data that leverages natural language to improve understanding, annotation, and cross-modal retrieval, significantly advancing human activity analysis and healthcare applications.

Contribution

It develops the largest sensor-language dataset and extends multimodal pretraining architectures to sensor data, enabling zero-shot, few-shot, and cross-modal tasks.

Findings

Outperforms state-of-the-art in zero-shot and few-shot recognition

Demonstrates effective sensor captioning and zero-shot generalization

Scales well with data and improves label efficiency

Abstract

We present SensorLM, a family of sensor-language foundation models that enable wearable sensor data understanding with natural language. Despite its pervasive nature, aligning and interpreting sensor data with language remains challenging due to the lack of paired, richly annotated sensor-text descriptions in uncurated, real-world wearable data. We introduce a hierarchical caption generation pipeline designed to capture statistical, structural, and semantic information from sensor data. This approach enabled the curation of the largest sensor-language dataset to date, comprising over 59.7 million hours of data from more than 103,000 people. Furthermore, SensorLM extends prominent multimodal pretraining architectures (e.g., CLIP, CoCa) and recovers them as specific variants within a generic architecture. Extensive experiments on real-world tasks in human activity analysis and healthcare verify the superior performance of SensorLM over state-of-the-art in zero-shot recognition, few-shot learning, and cross-modal retrieval. SensorLM also demonstrates intriguing capabilities including scaling behaviors, label efficiency, sensor captioning, and zero-shot generalization to unseen tasks.