DM$^3$T: Harmonizing Modalities via Diffusion for Multi-Object Tracking

TL;DR

DM$^3$T introduces a diffusion-inspired multimodal fusion framework for multi-object tracking, achieving superior accuracy by iteratively harmonizing features from visible and thermal modalities.

Contribution

The paper presents a novel diffusion-based approach for multimodal feature fusion in MOT, enabling deeper integration and improved tracking robustness over traditional methods.

Findings

Achieves 41.7 HOTA on VT-MOT benchmark, outperforming state-of-the-art.

Introduces a Cross-Modal Diffusion Fusion module for iterative feature alignment.

Employs a Diffusion Refiner to enhance feature representation.

Abstract

Multi-object tracking (MOT) is a fundamental task in computer vision with critical applications in autonomous driving and robotics. Multimodal MOT that integrates visible light and thermal infrared information is particularly essential for robust autonomous driving systems. However, effectively fusing these heterogeneous modalities is challenging. Simple strategies like concatenation or addition often fail to bridge the significant non-linear distribution gap between their feature representations, which can lead to modality conflicts and degrade tracking accuracy. Drawing inspiration from the connection between multimodal MOT and the iterative refinement in diffusion models, this paper proposes DM$^3$T, a novel framework that reformulates multimodal fusion as an iterative feature alignment process to generate accurate and temporally coherent object trajectories. Our approach performs iterative cross-modal harmonization through a proposed Cross-Modal Diffusion Fusion (C-MDF) module. In this process, features from both modalities provide mutual guidance, iteratively projecting them onto a shared, consistent feature manifold. This enables the learning of complementary information and achieves deeper fusion compared to conventional methods. Additionally, we introduce a plug-and-play Diffusion Refiner (DR) to enhance and refine the unified feature representation. To further improve tracking robustness, we design a Hierarchical Tracker that adaptively handles confidence estimation. DM$^3$T unifies object detection, state estimation, and data association into a comprehensive online tracking framework without complex post-processing. Extensive experiments on the VT-MOT benchmark demonstrate that our method achieves 41.7 HOTA, representing a 1.54% relative improvement over existing state-of-the-art methods. The code and models are available at https://vranlee.github.io/DM-3-T/.