IC3M: In-Car Multimodal Multi-object Monitoring for Abnormal Status of Both Driver and Passengers

TL;DR

IC3M is a multimodal in-car monitoring framework that improves abnormal status detection of drivers and passengers by addressing data scarcity, class imbalance, and missing modalities through adaptive pseudo-labeling and cross-modality reconstruction.

Contribution

The paper introduces IC3M, a novel multimodal framework with adaptive pseudo-labeling and missing modality reconstruction for robust in-car abnormal status monitoring.

Findings

Outperforms state-of-the-art benchmarks in accuracy, precision, and recall.

Demonstrates robustness under limited labeled data and missing modalities.

Effectively monitors both driver and passenger health conditions.

Abstract

Recently, in-car monitoring has emerged as a promising technology for detecting early-stage abnormal status of the driver and providing timely alerts to prevent traffic accidents. Although training models with multimodal data enhances the reliability of abnormal status detection, the scarcity of labeled data and the imbalance of class distribution impede the extraction of critical abnormal state features, significantly deteriorating training performance. Furthermore, missing modalities due to environment and hardware limitations further exacerbate the challenge of abnormal status identification. More importantly, monitoring abnormal health conditions of passengers, particularly in elderly care, is of paramount importance but remains underexplored. To address these challenges, we introduce our IC3M, an efficient camera-rotation-based multimodal framework for monitoring both driver and passengers in a car. Our IC3M comprises two key modules: an adaptive threshold pseudo-labeling strategy and a missing modality reconstruction. The former customizes pseudo-labeling thresholds for different classes based on the class distribution, generating class-balanced pseudo labels to guide model training effectively, while the latter leverages crossmodality relationships learned from limited labels to accurately recover missing modalities by distribution transferring from available modalities. Extensive experimental results demonstrate that IC3M outperforms state-of-the-art benchmarks in accuracy, precision, and recall while exhibiting superior robustness under limited labeled data and severe missing modality.