Seeing Roads Through Words: A Language-Guided Framework for RGB-T Driving Scene Segmentation

TL;DR

CLARITY is a dynamic, scene-adaptive RGB-T segmentation framework guided by vision-language priors, significantly improving robustness and accuracy in challenging driving conditions.

Contribution

It introduces a scene-aware fusion strategy guided by vision-language models, enhancing segmentation performance over static fusion methods.

Findings

Achieves new state-of-the-art performance on MFNet dataset.

Effectively preserves dark-object semantics in adverse conditions.

Enforces structural consistency to sharpen object boundaries.

Abstract

Robust semantic segmentation of road scenes under adverse illumination, lighting, and shadow conditions remain a core challenge for autonomous driving applications. RGB-Thermal fusion is a standard approach, yet existing methods apply static fusion strategies uniformly across all conditions, allowing modality-specific noise to propagate throughout the network. Hence, we propose CLARITY that dynamically adapts its fusion strategy to the detected scene condition. Guided by vision-language model (VLM) priors, the network learns to modulate each modality's contribution based on the illumination state while leveraging object embeddings for segmentation, rather than applying a fixed fusion policy. We further introduce two mechanisms, i.e., one which preserves valid dark-object semantics that prior noise-suppression methods incorrectly discard, and a hierarchical decoder that enforces structural consistency across scales to sharpen boundaries on thin objects. Experiments on the MFNet dataset demonstrate that CLARITY establishes a new state-of-the-art (SOTA), achieving 62.3% mIoU and 77.5% mAcc.