CM2-Net: Continual Cross-Modal Mapping Network for Driver Action Recognition

TL;DR

This paper introduces CM2-Net, a continual learning framework that effectively integrates new non-RGB modalities for driver action recognition by using instructive prompts from previously learned modalities, improving recognition accuracy.

Contribution

The paper proposes a novel Continual Cross-Modal Mapping Network with Accumulative Cross-modal Mapping Prompting to enhance multi-modal driver action recognition in continual learning settings.

Findings

CM2-Net outperforms existing methods on the Drive&Act dataset.

The approach effectively leverages prompts to incorporate new modalities.

Results show improved recognition accuracy for both uni- and multi-modal data.

Abstract

Driver action recognition has significantly advanced in enhancing driver-vehicle interactions and ensuring driving safety by integrating multiple modalities, such as infrared and depth. Nevertheless, compared to RGB modality only, it is always laborious and costly to collect extensive data for all types of non-RGB modalities in car cabin environments. Therefore, previous works have suggested independently learning each non-RGB modality by fine-tuning a model pre-trained on RGB videos, but these methods are less effective in extracting informative features when faced with newly-incoming modalities due to large domain gaps. In contrast, we propose a Continual Cross-Modal Mapping Network (CM2-Net) to continually learn each newly-incoming modality with instructive prompts from the previously-learned modalities. Specifically, we have developed Accumulative Cross-modal Mapping Prompting (ACMP), to map the discriminative and informative features learned from previous modalities into the feature space of newly-incoming modalities. Then, when faced with newly-incoming modalities, these mapped features are able to provide effective prompts for which features should be extracted and prioritized. These prompts are accumulating throughout the continual learning process, thereby boosting further recognition performances. Extensive experiments conducted on the Drive&Act dataset demonstrate the performance superiority of CM2-Net on both uni- and multi-modal driver action recognition.