Babel: A Scalable Pre-trained Model for Multi-Modal Sensing via Expandable Modality Alignment

TL;DR

Babel is a scalable, expandable multi-modal sensing model that effectively aligns multiple sensing modalities, overcoming data scarcity and partial pairing challenges, to enhance human activity recognition and enable new sensing applications.

Contribution

The paper introduces the concept of expandable modality alignment, transforming multi-modality alignment into binary alignments, with novel techniques to handle data scarcity and modality integration.

Findings

Achieves up to 22% accuracy improvement in multi-modal sensing tasks.

Effectively aligns six sensing modalities including Wi-Fi, mmWave, IMU, LiDAR, video, and depth.

Enables cross-modality retrieval and sensing comprehension through case studies.

Abstract

This paper presents Babel, the expandable modality alignment model, specially designed for multi-modal sensing. While there has been considerable work on multi-modality alignment, they all struggle to effectively incorporate multiple sensing modalities due to the data scarcity constraints. How to utilize multi-modal data with partial pairings in sensing remains an unresolved challenge. Babel tackles this challenge by introducing the concept of expandable modality alignment. The key idea involves transforming the N-modality alignment into a series of binary-modality alignments. Novel techniques are also proposed to further mitigate data scarcity issue and balance the contribution of the newly incorporated modality with the previously established modality alignment during the expandable alignment process. We provide the comprehensive implementation. In the pre-training phase, Babel currently aligns 6 sensing modalities, namely Wi-Fi, mmWave, IMU, LiDAR, video, and depth. For the deployment phase, as a foundation model, any single or combination of aligned modalities could be selected from Babel and applied to downstream tasks. Evaluation demonstrates Babel's outstanding performance on eight human activity recognition datasets, compared to a broad range of baselines e.g., the SOTA single-modal sensing networks, multi-modal sensing framework, and multi-modal large language models. Babel not only improves the performance of individual modality sensing (12% averaged accuracy improvement), but also effectively fuses multiple available modalities (up to 22% accuracy increase). Case studies also highlight emerging application scenarios empowered by Babel, including cross-modality retrieval (i.e., sensing imaging), and bridging LLM for sensing comprehension.