ATAS: Any-to-Any Self-Distillation for Enhanced Open-Vocabulary Dense Prediction

TL;DR

ATAS introduces a self-distillation method that improves CLIP's fine-grained, region-level understanding in open-vocabulary dense prediction tasks by enhancing semantic coherence and alignment without extra supervision.

Contribution

The paper proposes ATAS, a novel self-distillation approach that refines CLIP's representations for better dense prediction performance using only unlabeled images.

Findings

Significant performance improvements on object detection benchmarks.

Outperforms baseline CLIP models in semantic segmentation.

Effectively maintains semantic coherence while sharpening local details.

Abstract

Vision-language models such as CLIP have recently propelled open-vocabulary dense prediction tasks by enabling recognition of a broad range of visual concepts. However, CLIP still struggles with fine-grained, region-level understanding, hindering its effectiveness on these dense prediction tasks. We identify two pivotal factors required to address this limitation: semantic coherence and fine-grained vision-language alignment. Current adaptation methods often improve fine-grained alignment at the expense of semantic coherence, and often rely on extra modules or supervised fine-tuning. To overcome these issues, we propose Any-to-Any Self-Distillation (ATAS), a novel approach that simultaneously enhances semantic coherence and fine-grained alignment by leveraging own knowledge of a model across all representation levels. Unlike prior methods, ATAS uses only unlabeled images and an internal self-distillation process to refine representations of CLIP vision encoders, preserving local semantic consistency while sharpening local detail recognition. On open-vocabulary object detection and semantic segmentation benchmarks, ATAS achieves substantial performance gains, outperforming baseline CLIP models. These results validate the effectiveness of our approach and underscore the importance of jointly maintaining semantic coherence and fine-grained alignment for advanced open-vocabulary dense prediction.