Correspondence as Video: Test-Time Adaption on SAM2 for Reference Segmentation in the Wild

TL;DR

This paper introduces CAV-SAM, a lightweight test-time adaptation method that models reference-target image pairs as pseudo videos, enabling SAM2 to improve reference segmentation in the wild without extensive meta-training.

Contribution

It proposes a novel pseudo video perspective for reference segmentation, utilizing diffusion models and test-time fine-tuning to adapt SAM2 efficiently in real-world scenarios.

Findings

Achieved over 5% improvement over SOTA methods on benchmark datasets.

Introduced a diffusion-based semantic transition module for better semantic understanding.

Developed a test-time geometric alignment module for precise adaptation.

Abstract

Large vision models like the Segment Anything Model (SAM) exhibit significant limitations when applied to downstream tasks in the wild. Consequently, reference segmentation, which leverages reference images and their corresponding masks to impart novel knowledge to the model, emerges as a promising new direction for adapting vision models. However, existing reference segmentation approaches predominantly rely on meta-learning, which still necessitates an extensive meta-training process and brings massive data and computational cost. In this study, we propose a novel approach by representing the inherent correspondence between reference-target image pairs as a pseudo video. This perspective allows the latest version of SAM, known as SAM2, which is equipped with interactive video object segmentation (iVOS) capabilities, to be adapted to downstream tasks in a lightweight manner. We term this approach Correspondence As Video for SAM (CAV-SAM). CAV-SAM comprises two key modules: the Diffusion-Based Semantic Transition (DBST) module employs a diffusion model to construct a semantic transformation sequence, while the Test-Time Geometric Alignment (TTGA) module aligns the geometric changes within this sequence through test-time fine-tuning. We evaluated CAVSAM on widely-used datasets, achieving segmentation performance improvements exceeding 5% over SOTA methods. Implementation is provided in the supplementary materials.