Interpretable Open-Vocabulary Referring Object Detection with Reverse Contrast Attention

TL;DR

This paper introduces Reverse Contrast Attention (RCA), a simple plug-in that improves object localization in vision-language transformers by reweighting attention, leading to better open-vocabulary referring object detection without retraining.

Contribution

The paper presents RCA, a novel attention reweighting method that enhances localization and interpretability in vision-language models for open-vocabulary detection tasks.

Findings

RCA improves FitAP in 11 out of 15 models, with gains up to +26.6%.

Effectiveness correlates with attention sharpness and fusion timing.

Late-fusion models benefit consistently from RCA.

Abstract

We propose Reverse Contrast Attention (RCA), a plug-in method that enhances object localization in vision-language transformers without retraining. RCA reweights final-layer attention by suppressing extremes and amplifying mid-level activations to let semantically relevant but subdued tokens guide predictions. We evaluate it on Open Vocabulary Referring Object Detection (OV-RefOD), introducing FitAP, a confidence-free average precision metric based on IoU and box area. RCA improves FitAP in 11 out of 15 open-source VLMs, with gains up to $+26.6\%$. Effectiveness aligns with attention sharpness and fusion timing; while late-fusion models benefit consistently, models like $\texttt{DeepSeek-VL2}$ also improve, pointing to capacity and disentanglement as key factors. RCA offers both interpretability and performance gains for multimodal transformers. Codes and dataset are available from https://github.com/earl-juanico/rca