TEM^3-Learning: Time-Efficient Multimodal Multi-Task Learning for Advanced Assistive Driving

TL;DR

TEM^3-Learning introduces a time-efficient, multimodal multi-task framework for assistive driving that jointly recognizes driver emotions, behaviors, traffic context, and vehicle actions with high accuracy and real-time performance.

Contribution

The paper presents a novel two-stage architecture combining efficient feature extraction and adaptive multimodal integration for multi-task assistive driving, addressing limitations of modality constraints and computational inefficiency.

Findings

Achieves state-of-the-art accuracy on AIDE dataset for all four tasks.

Maintains a lightweight model with fewer than 6 million parameters.

Delivers 142.32 FPS inference speed, enabling real-time deployment.

Abstract

Multi-task learning (MTL) can advance assistive driving by exploring inter-task correlations through shared representations. However, existing methods face two critical limitations: single-modality constraints limiting comprehensive scene understanding and inefficient architectures impeding real-time deployment. This paper proposes TEM^3-Learning (Time-Efficient Multimodal Multi-task Learning), a novel framework that jointly optimizes driver emotion recognition, driver behavior recognition, traffic context recognition, and vehicle behavior recognition through a two-stage architecture. The first component, the mamba-based multi-view temporal-spatial feature extraction subnetwork (MTS-Mamba), introduces a forward-backward temporal scanning mechanism and global-local spatial attention to efficiently extract low-cost temporal-spatial features from multi-view sequential images. The second component, the MTL-based gated multimodal feature integrator (MGMI), employs task-specific multi-gating modules to adaptively highlight the most relevant modality features for each task, effectively alleviating the negative transfer problem in MTL. Evaluation on the AIDE dataset, our proposed model achieves state-of-the-art accuracy across all four tasks, maintaining a lightweight architecture with fewer than 6 million parameters and delivering an impressive 142.32 FPS inference speed. Rigorous ablation studies further validate the effectiveness of the proposed framework and the independent contributions of each module. The code is available on https://github.com/Wenzhuo-Liu/TEM3-Learning.