MCAM: Multimodal Causal Analysis Model for Ego-Vehicle-Level Driving Video Understanding

TL;DR

MCAM introduces a multimodal causal analysis framework for autonomous driving videos, effectively modeling causal relationships across visual and language data to improve understanding and reasoning.

Contribution

The paper presents a novel causal analysis model that constructs latent causal structures across modalities, addressing limitations of existing methods in driving video understanding.

Findings

Achieves state-of-the-art performance on BDD-X and CoVLA datasets.

Effectively models causal relationships in driving videos.

Demonstrates superior causal feature alignment across modalities.

Abstract

Accurate driving behavior recognition and reasoning are critical for autonomous driving video understanding. However, existing methods often tend to dig out the shallow causal, fail to address spurious correlations across modalities, and ignore the ego-vehicle level causality modeling. To overcome these limitations, we propose a novel Multimodal Causal Analysis Model (MCAM) that constructs latent causal structures between visual and language modalities. Firstly, we design a multi-level feature extractor to capture long-range dependencies. Secondly, we design a causal analysis module that dynamically models driving scenarios using a directed acyclic graph (DAG) of driving states. Thirdly, we utilize a vision-language transformer to align critical visual features with their corresponding linguistic expressions. Extensive experiments on the BDD-X, and CoVLA datasets demonstrate that MCAM achieves SOTA performance in visual-language causal relationship learning. Furthermore, the model exhibits superior capability in capturing causal characteristics within video sequences, showcasing its effectiveness for autonomous driving applications. The code is available at https://github.com/SixCorePeach/MCAM.