T-Rex-Omni: Integrating Negative Visual Prompt in Generic Object Detection

TL;DR

T-Rex-Omni introduces a framework that incorporates negative visual prompts into open-set object detection, effectively reducing false positives from distractors and improving zero-shot detection performance especially in long-tailed scenarios.

Contribution

It presents a novel negative prompt integration method with a unified encoder, a training-free negation module, and a discriminative loss, advancing open-set detection beyond positive-only paradigms.

Findings

Achieves state-of-the-art zero-shot detection performance.

Significantly narrows gap between visual and text-prompted methods.

Excels in long-tailed detection scenarios, e.g., 51.2 AP_r on LVIS-minival.

Abstract

Object detection methods have evolved from closed-set to open-set paradigms over the years. Current open-set object detectors, however, remain constrained by their exclusive reliance on positive indicators based on given prompts like text descriptions or visual exemplars. This positive-only paradigm experiences consistent vulnerability to visually similar but semantically different distractors. We propose T-Rex-Omni, a novel framework that addresses this limitation by incorporating negative visual prompts to negate hard negative distractors. Specifically, we first introduce a unified visual prompt encoder that jointly processes positive and negative visual prompts. Next, a training-free Negating Negative Computing (NNC) module is proposed to dynamically suppress negative responses during the probability computing stage. To further boost performance through fine-tuning, our Negating Negative Hinge (NNH) loss enforces discriminative margins between positive and negative embeddings. T-Rex-Omni supports flexible deployment in both positive-only and joint positive-negative inference modes, accommodating either user-specified or automatically generated negative examples. Extensive experiments demonstrate remarkable zero-shot detection performance, significantly narrowing the performance gap between visual-prompted and text-prompted methods while showing particular strength in long-tailed scenarios (51.2 AP_r on LVIS-minival). This work establishes negative prompts as a crucial new dimension for advancing open-set visual recognition systems.