DC-TTA: Divide-and-Conquer Framework for Test-Time Adaptation of Interactive Segmentation

TL;DR

DC-TTA introduces a divide-and-conquer test-time adaptation framework for interactive segmentation, improving SAM's performance in complex scenarios by locally adapting models based on user interactions.

Contribution

It proposes a novel divide-and-conquer TTA approach that partitions user cues for localized model updates, enhancing segmentation accuracy in challenging cases.

Findings

DC-TTA outperforms zero-shot SAM and conventional TTA methods.

It achieves better results with fewer user interactions.

Effective in complex tasks like camouflaged object segmentation.

Abstract

Interactive segmentation (IS) allows users to iteratively refine object boundaries with minimal cues, such as positive and negative clicks. While the Segment Anything Model (SAM) has garnered attention in the IS community for its promptable segmentation capabilities, it often struggles in specialized domains or when handling complex scenarios (e.g., camouflaged or multi-part objects). To overcome these challenges, we propose DC-TTA, a novel test-time adaptation (TTA) framework that adapts SAM on a per-sample basis by leveraging user interactions as supervision. Instead of forcing a single model to incorporate all user clicks at once, DC-TTA partitions the clicks into more coherent subsets, each processed independently via TTA with a separated model. This Divide-and-Conquer strategy reduces conflicts among diverse cues and enables more localized updates. Finally, we merge the adapted models to form a unified predictor that integrates the specialized knowledge from each subset. Experimental results across various benchmarks demonstrate that DC-TTA significantly outperforms SAM's zero-shot results and conventional TTA methods, effectively handling complex tasks such as camouflaged object segmentation with fewer interactions and improved accuracy.