Generalized Decoupled Learning for Enhancing Open-Vocabulary Dense Perception

TL;DR

This paper introduces DeCLIP, a framework that improves open-vocabulary dense perception by decoupling CLIP's features and enhancing them with semantic and spatial cues, achieving state-of-the-art results across multiple tasks.

Contribution

DeCLIP decouples CLIP's self-attention to separately enhance content and context features, integrating semantic and spatial information for better dense perception.

Findings

Achieves state-of-the-art performance on 2D detection and segmentation.

Improves 3D instance segmentation and video instance segmentation.

Enhances 6D object pose estimation accuracy.

Abstract

Dense visual perception tasks have been constrained by their reliance on predefined categories, limiting their applicability in real-world scenarios where visual concepts are unbounded. While Vision-Language Models (VLMs) like CLIP have shown promise in open-vocabulary tasks, their direct application to dense perception often leads to suboptimal performance due to limitations in local feature representation. In this work, we present our observation that CLIP's image tokens struggle to effectively aggregate information from spatially or semantically related regions, resulting in features that lack local discriminability and spatial consistency. To address this issue, we propose DeCLIP, a novel framework that enhances CLIP by decoupling the self-attention module to obtain ``content'' and ``context'' features respectively. \revise{The context features are enhanced by jointly distilling semantic correlations from Vision Foundation Models (VFMs) and object integrity cues from diffusion models, thereby enhancing spatial consistency. In parallel, the content features are aligned with image crop representations and constrained by region correlations from VFMs to improve local discriminability. Extensive experiments demonstrate that DeCLIP establishes a solid foundation for open-vocabulary dense perception, consistently achieving state-of-the-art performance across a broad spectrum of tasks, including 2D detection and segmentation, 3D instance segmentation, video instance segmentation, and 6D object pose estimation.} Code is available at https://github.com/xiaomoguhz/DeCLIP