Fusing in 3D: Free-Viewpoint Fusion Rendering with a 3D Infrared-Visible Scene Representation

TL;DR

This paper introduces a novel 3D scene representation and rendering framework for infrared-visible image fusion, enabling comprehensive scene understanding and high-quality fused images from multiple viewpoints.

Contribution

It proposes a 3D scene reconstruction and fusion method that addresses limitations of 2D fusion techniques by incorporating scene geometry and cross-modal adjustments.

Findings

Effective scene reconstruction from multimodal inputs

Improved fusion quality with preserved modality features

Robustness across complex scenarios

Abstract

Infrared-visible image fusion aims to integrate infrared and visible information into a single fused image. Existing 2D fusion methods focus on fusing images from fixed camera viewpoints, neglecting a comprehensive understanding of complex scenarios, which results in the loss of critical information about the scene. To address this limitation, we propose a novel Infrared-Visible Gaussian Fusion (IVGF) framework, which reconstructs scene geometry from multimodal 2D inputs and enables direct rendering of fused images. Specifically, we propose a cross-modal adjustment (CMA) module that modulates the opacity of Gaussians to solve the problem of cross-modal conflicts. Moreover, to preserve the distinctive features from both modalities, we introduce a fusion loss that guides the optimization of CMA, thus ensuring that the fused image retains the critical characteristics of each modality. Comprehensive qualitative and quantitative experiments demonstrate the effectiveness of the proposed method.