VLC Fusion: Vision-Language Conditioned Sensor Fusion for Robust Object Detection

TL;DR

VLC Fusion introduces a novel sensor fusion framework that uses a vision-language model to adaptively weight sensor modalities based on environmental context, significantly improving object detection robustness across diverse conditions.

Contribution

The paper proposes a new fusion method leveraging a vision-language model to dynamically adjust sensor weights according to environmental cues, enhancing detection performance.

Findings

Outperforms traditional fusion methods in real-world datasets

Improves detection accuracy in varied environmental conditions

Effective across multiple sensor modalities

Abstract

Although fusing multiple sensor modalities can enhance object detection performance, existing fusion approaches often overlook subtle variations in environmental conditions and sensor inputs. As a result, they struggle to adaptively weight each modality under such variations. To address this challenge, we introduce Vision-Language Conditioned Fusion (VLC Fusion), a novel fusion framework that leverages a Vision-Language Model (VLM) to condition the fusion process on nuanced environmental cues. By capturing high-level environmental context such as as darkness, rain, and camera blurring, the VLM guides the model to dynamically adjust modality weights based on the current scene. We evaluate VLC Fusion on real-world autonomous driving and military target detection datasets that include image, LIDAR, and mid-wave infrared modalities. Our experiments show that VLC Fusion consistently outperforms conventional fusion baselines, achieving improved detection accuracy in both seen and unseen scenarios.