Cross-Domain Few-Shot Semantic Segmentation via Doubly Matching Transformation

TL;DR

This paper introduces DMTNet, a novel approach for cross-domain few-shot semantic segmentation that uses query-specific transformations and self-finetuning to improve generalization across domains with limited support images.

Contribution

The paper proposes a Doubly Matching Transformation network with self-matching and dual hypercorrelation modules, and a test-time self-finetuning strategy, to enhance cross-domain segmentation performance.

Findings

DMTNet outperforms state-of-the-art methods on four datasets.

Query-specific transformations reduce overfitting in few-shot scenarios.

Self-finetuning improves segmentation accuracy in unseen domains.

Abstract

Cross-Domain Few-shot Semantic Segmentation (CD-FSS) aims to train generalized models that can segment classes from different domains with a few labeled images. Previous works have proven the effectiveness of feature transformation in addressing CD-FSS. However, they completely rely on support images for feature transformation, and repeatedly utilizing a few support images for each class may easily lead to overfitting and overlooking intra-class appearance differences. In this paper, we propose a Doubly Matching Transformation-based Network (DMTNet) to solve the above issue. Instead of completely relying on support images, we propose Self-Matching Transformation (SMT) to construct query-specific transformation matrices based on query images themselves to transform domain-specific query features into domain-agnostic ones. Calculating query-specific transformation matrices can prevent overfitting, especially for the meta-testing stage where only one or several images are used as support images to segment hundreds or thousands of images. After obtaining domain-agnostic features, we exploit a Dual Hypercorrelation Construction (DHC) module to explore the hypercorrelations between the query image with the foreground and background of the support image, based on which foreground and background prediction maps are generated and supervised, respectively, to enhance the segmentation result. In addition, we propose a Test-time Self-Finetuning (TSF) strategy to more accurately self-tune the query prediction in unseen domains. Extensive experiments on four popular datasets show that DMTNet achieves superior performance over state-of-the-art approaches. Code is available at https://github.com/ChenJiayi68/DMTNet.