HoloLLM: Multisensory Foundation Model for Language-Grounded Human Sensing and Reasoning

TL;DR

HoloLLM is a multisensory foundation model that integrates diverse sensing modalities like LiDAR, infrared, mmWave radar, and WiFi to enhance human perception and reasoning in embodied agents, overcoming visual limitations.

Contribution

The paper introduces HoloLLM, a novel multimodal large language model that incorporates uncommon sensors and a universal modality-injection mechanism for improved multisensory perception.

Findings

Outperforms existing models on new benchmarks by up to 30% in sensing accuracy.

Effectively integrates heterogeneous sensor data without significant alignment overhead.

Demonstrates robustness in real-world scenarios with occlusions and poor lighting.

Abstract

Embodied agents operating in smart homes must understand human behavior through diverse sensory inputs and communicate via natural language. While Vision-Language Models (VLMs) have enabled impressive language-grounded perception, their reliance on visual data limits robustness in real-world scenarios with occlusions, poor lighting, or privacy constraints. In this paper, we introduce HoloLLM, a Multimodal Large Language Model (MLLM) that integrates uncommon but powerful sensing modalities, such as LiDAR, infrared, mmWave radar, and WiFi, to enable seamless human perception and reasoning across heterogeneous environments. We address two key challenges: (1) the scarcity of aligned modality-text data for rare sensors, and (2) the heterogeneity of their physical signal representations. To overcome these, we design a Universal Modality-Injection Projector (UMIP) that enhances pre-aligned modality embeddings with fine-grained, text-aligned features from tailored encoders via coarse-to-fine cross-attention without introducing significant alignment overhead. We further introduce a human-VLM collaborative data curation pipeline to generate paired textual annotations for sensing datasets. Extensive experiments on two newly constructed benchmarks show that HoloLLM significantly outperforms existing MLLMs, improving language-grounded human sensing accuracy by up to 30%. This work establishes a new foundation for real-world, language-informed multisensory embodied intelligence.