Set Pivot Learning: Redefining Generalized Segmentation with Vision Foundation Models

TL;DR

This paper introduces Set Pivot Learning, a new paradigm for domain generalization using Vision Foundation Models, emphasizing adaptive refinement and VFM-centric tuning to improve generalized segmentation performance.

Contribution

The paper proposes Set Pivot Learning as a novel domain generalization framework leveraging VFMs, with a dynamic prompt fine-tuning method for enhanced segmentation results.

Findings

Outperforms state-of-the-art methods on benchmark datasets.

Demonstrates improved robustness in generalized segmentation.

Validates effectiveness of dynamic prompt fine-tuning.

Abstract

In this paper, we introduce, for the first time, the concept of Set Pivot Learning, a paradigm shift that redefines domain generalization (DG) based on Vision Foundation Models (VFMs). Traditional DG assumes that the target domain is inaccessible during training, but the emergence of VFMs, trained on vast and diverse data, renders this assumption unclear and obsolete. Traditional DG assumes that the target domain is inaccessible during training, but the emergence of VFMs, which are trained on vast and diverse datasets, renders this assumption unclear and obsolete. To address this challenge, we propose Set Pivot Learning (SPL), a new definition of domain migration task based on VFMs, which is more suitable for current research and application requirements. Unlike conventional DG methods, SPL prioritizes adaptive refinement over rigid domain transfer, ensuring continuous alignment with evolving real-world conditions. Specifically, SPL features two key attributes: (i) Dynamic adaptation, transitioning from static domain alignment to flexible, task-driven feature optimization, enabling models to evolve with downstream scenarios; (ii) VFM-centric tuning, leveraging pretrained knowledge as a pivot to hone task-specific representations while preserving cross-domain robustness. Building on SPL, we propose a Dynamic Prompt Fine-Tuning method, which combines a Dynamic Class-aware Prompter with a Prompt-guided Feature Focuser, to elevate VFM performance in targeted scenarios. Extensive experiments on benchmark datasets show the effectiveness of our method, highlighting its superiority over state-of-the-art methods, particularly in generalized segmentation.