Visual Modality Prompt for Adapting Vision-Language Object Detectors

TL;DR

This paper introduces ModPrompt, a visual prompt method that adapts vision-language object detectors to new modalities like infrared and depth without losing zero-shot performance, outperforming traditional fine-tuning.

Contribution

The paper proposes ModPrompt, a novel visual prompt strategy with an encoder-decoder design and residuals, enabling effective modality adaptation for vision-language detectors without degrading zero-shot capabilities.

Findings

Achieves comparable performance to full fine-tuning on infrared and depth datasets.

Preserves zero-shot detection capabilities after modality adaptation.

Effective on YOLO-World and Grounding DINO detectors.

Abstract

The zero-shot performance of object detectors degrades when tested on different modalities, such as infrared and depth. While recent work has explored image translation techniques to adapt detectors to new modalities, these methods are limited to a single modality and apply only to traditional detectors. Recently, vision-language detectors, such as YOLO-World and Grounding DINO, have shown promising zero-shot capabilities, however, they have not yet been adapted for other visual modalities. Traditional fine-tuning approaches compromise the zero-shot capabilities of the detectors. The visual prompt strategies commonly used for classification with vision-language models apply the same linear prompt translation to each image, making them less effective. To address these limitations, we propose ModPrompt, a visual prompt strategy to adapt vision-language detectors to new modalities without degrading zero-shot performance. In particular, an encoder-decoder visual prompt strategy is proposed, further enhanced by the integration of inference-friendly modality prompt decoupled residual, facilitating a more robust adaptation. Empirical benchmarking results show our method for modality adaptation on two vision-language detectors, YOLO-World and Grounding DINO, and on challenging infrared (LLVIP, FLIR) and depth (NYUv2) datasets, achieving performance comparable to full fine-tuning while preserving the model's zero-shot capability. Code available at: https://github.com/heitorrapela/ModPrompt.