From Words to Wavelengths: VLMs for Few-Shot Multispectral Object Detection

TL;DR

This paper investigates the use of Vision-Language Models (VLMs) for few-shot multispectral object detection, demonstrating their effectiveness in data-scarce scenarios and their ability to transfer semantic knowledge across spectral modalities.

Contribution

It adapts two VLM-based detectors for multispectral inputs and introduces a mechanism to integrate multiple modalities, advancing data-efficient multispectral perception.

Findings

VLM-based detectors outperform specialized models in few-shot settings

They achieve competitive results in fully supervised scenarios

Semantic priors from VLMs transfer effectively across spectral modalities

Abstract

Multispectral object detection is critical for safety-sensitive applications such as autonomous driving and surveillance, where robust perception under diverse illumination conditions is essential. However, the limited availability of annotated multispectral data severely restricts the training of deep detectors. In such data-scarce scenarios, textual class information can serve as a valuable source of semantic supervision. Motivated by the recent success of Vision-Language Models (VLMs) in computer vision, we explore their potential for few-shot multispectral object detection. Specifically, we adapt two representative VLM-based detectors, Grounding DINO and YOLO-World, to handle multispectral inputs and propose an effective mechanism to integrate text, visual and thermal modalities. Through extensive experiments on two popular multispectral image benchmarks, FLIR and M3FD, we demonstrate that VLM-based detectors not only excel in few-shot regimes, significantly outperforming specialized multispectral models trained with comparable data, but also achieve competitive or superior results under fully supervised settings. Our findings reveal that the semantic priors learned by large-scale VLMs effectively transfer to unseen spectral modalities, ofFering a powerful pathway toward data-efficient multispectral perception.