TemCoCo: Temporally Consistent Multi-modal Video Fusion with Visual-Semantic Collaboration

TL;DR

TemCoCo introduces a novel video fusion framework that explicitly models temporal dependencies and visual-semantic collaboration, significantly improving temporal consistency and fusion quality in multi-modal videos.

Contribution

It is the first to incorporate explicit temporal modeling with visual-semantic collaboration in video fusion, enhancing consistency and semantic accuracy.

Findings

Outperforms existing methods on public datasets.

Achieves higher temporal consistency scores.

Demonstrates improved visual and semantic fidelity.

Abstract

Existing multi-modal fusion methods typically apply static frame-based image fusion techniques directly to video fusion tasks, neglecting inherent temporal dependencies and leading to inconsistent results across frames. To address this limitation, we propose the first video fusion framework that explicitly incorporates temporal modeling with visual-semantic collaboration to simultaneously ensure visual fidelity, semantic accuracy, and temporal consistency. First, we introduce a visual-semantic interaction module consisting of a semantic branch and a visual branch, with Dinov2 and VGG19 employed for targeted distillation, allowing simultaneous enhancement of both the visual and semantic representations. Second, we pioneer integrate the video degradation enhancement task into the video fusion pipeline by constructing a temporal cooperative module, which leverages temporal dependencies to facilitate weak information recovery. Third, to ensure temporal consistency, we embed a temporal-enhanced mechanism into the network and devise a temporal loss to guide the optimization process. Finally, we introduce two innovative evaluation metrics tailored for video fusion, aimed at assessing the temporal consistency of the generated fused videos. Extensive experimental results on public video datasets demonstrate the superiority of our method. Our code is released at https://github.com/Meiqi-Gong/TemCoCo.