# Plug-in Feedback Self-adaptive Attention in CLIP for Training-free Open-Vocabulary Segmentation

## TL;DR

This paper introduces a training-free, feedback-driven self-adaptive framework that enhances open-vocabulary segmentation in CLIP by propagating output-based semantic cues back to intermediate attention, improving localization and coherence.

## Contribution

It proposes a novel plug-in, feedback-based method that improves CLIP's segmentation without additional training, applicable across multiple architectures and attention types.

## Key findings

- Consistent performance improvements across eight benchmarks.
- Effective integration with four state-of-the-art approaches.
- Enhancement of semantic coherence between predictions and internal representations.

## Abstract

CLIP exhibits strong visual-textual alignment but struggle with open-vocabulary segmentation due to poor localization. Prior methods enhance spatial coherence by modifying intermediate attention. But, this coherence isn't consistently propagated to the final output due to subsequent operations such as projections. Additionally, intermediate attention lacks direct interaction with text representations, such semantic discrepancy limits the full potential of CLIP. In this work, we propose a training-free, feedback-driven self-adaptive framework that adapts output-based patch-level correspondences back to the intermediate attention. The output predictions, being the culmination of the model's processing, encapsulate the most comprehensive visual and textual semantics about each patch. Our approach enhances semantic consistency between internal representations and final predictions by leveraging the model's outputs as a stronger spatial coherence prior. We design key modules, including attention isolation, confidence-based pruning for sparse adaptation, and adaptation ensemble, to effectively feedback the output coherence cues. Our method functions as a plug-in module, seamlessly integrating into four state-of-the-art approaches with three backbones (ViT-B, ViT-L, ViT-H). We further validate our framework across multiple attention types (Q-K, self-self, and Proxy augmented with MAE, SAM, and DINO). Our approach consistently improves their performance across eight benchmarks.