PhysLLM: Harnessing Large Language Models for Cross-Modal Remote Physiological Sensing

TL;DR

PhysLLM introduces a novel framework combining large language models with domain-specific components to improve remote physiological sensing accuracy and robustness under challenging conditions.

Contribution

The paper proposes PhysLLM, a collaborative optimization framework that aligns physiological signals with language models and introduces new algorithms for signal stability and cross-modal learning.

Findings

Achieves state-of-the-art accuracy on benchmark datasets.

Demonstrates robustness to illumination changes and motion artifacts.

Effectively integrates visual and textual information for physiological measurement.

Abstract

Remote photoplethysmography (rPPG) enables non-contact physiological measurement but remains highly susceptible to illumination changes, motion artifacts, and limited temporal modeling. Large Language Models (LLMs) excel at capturing long-range dependencies, offering a potential solution but struggle with the continuous, noise-sensitive nature of rPPG signals due to their text-centric design. To bridge this gap, we introduce the PhysLLM, a collaborative optimization framework that synergizes LLMs with domain-specific rPPG components. Specifically, the Text Prototype Guidance (TPG) strategy is proposed to establish cross-modal alignment by projecting hemodynamic features into LLM-interpretable semantic space, effectively bridging the representational gap between physiological signals and linguistic tokens. Besides, a novel Dual-Domain Stationary (DDS) Algorithm is proposed for resolving signal instability through adaptive time-frequency domain feature re-weighting. Finally, rPPG task-specific cues systematically inject physiological priors through physiological statistics, environmental contextual answering, and task description, leveraging cross-modal learning to integrate both visual and textual information, enabling dynamic adaptation to challenging scenarios like variable illumination and subject movements. Evaluation on four benchmark datasets, PhysLLM achieves state-of-the-art accuracy and robustness, demonstrating superior generalization across lighting variations and motion scenarios. The source code is available at https://github.com/Alex036225/PhysLLM.