TASeg: Text-aware RGB-T Semantic Segmentation based on Fine-tuning Vision Foundation Models

TL;DR

TASeg introduces a novel framework that combines fine-tuned vision foundation models with text embeddings to improve RGB-T semantic segmentation, especially in challenging environments.

Contribution

The paper presents a new text-aware RGB-T segmentation method using LoRA fine-tuning, a dynamic feature fusion module, and CLIP text embeddings to enhance semantic accuracy.

Findings

Outperforms existing methods on multiple datasets.

Requires fewer trainable parameters.

Effectively integrates textual and visual information.

Abstract

Reliable semantic segmentation of open environments is essential for intelligent systems, yet significant problems remain: 1) Existing RGB-T semantic segmentation models mainly rely on low-level visual features and lack high-level textual information, which struggle with accurate segmentation when categories share similar visual characteristics. 2) While SAM excels in instance-level segmentation, integrating it with thermal images and text is hindered by modality heterogeneity and computational inefficiency. To address these, we propose TASeg, a text-aware RGB-T segmentation framework by using Low-Rank Adaptation (LoRA) fine-tuning technology to adapt vision foundation models. Specifically, we propose a Dynamic Feature Fusion Module (DFFM) in the image encoder, which effectively merges features from multiple visual modalities while freezing SAM's original transformer blocks. Additionally, we incorporate CLIP-generated text embeddings in the mask decoder to enable semantic alignment, which further rectifies the classification error and improves the semantic understanding accuracy. Experimental results across diverse datasets demonstrate that our method achieves superior performance in challenging scenarios with fewer trainable parameters.