Shape-of-You: Fused Gromov-Wasserstein Optimal Transport for Semantic Correspondence in-the-Wild

TL;DR

Shape-of-You introduces a novel approach using Fused Gromov-Wasserstein optimal transport to improve semantic correspondence in challenging in-the-wild images by jointly considering feature similarity and structural consistency.

Contribution

It reformulates pseudo-label generation as a Fused Gromov-Wasserstein problem and proposes an efficient approximation method, enabling robust unsupervised learning of semantic correspondence.

Findings

Achieves state-of-the-art results on SPair-71k and AP-10k datasets.

Establishes a new benchmark in semantic correspondence without explicit geometric annotations.

Demonstrates robustness to structural ambiguities in in-the-wild images.

Abstract

Semantic correspondence is essential for handling diverse in-the-wild images lacking explicit correspondence annotations. While recent 2D foundation models offer powerful features, adapting them for unsupervised learning via nearest-neighbor pseudo-labels has key limitations: it operates locally, ignoring structural relationships, and consequently its reliance on 2D appearance fails to resolve geometric ambiguities arising from symmetries or repetitive features. In this work, we address this by reformulating pseudo-label generation as a Fused Gromov-Wasserstein (FGW) problem, which jointly optimizes inter-feature similarity and intra-structural consistency. Our framework, Shape-of-You (SoY), leverages a 3D foundation model to define this intra-structure in the geometric space, resolving abovementioned ambiguity. However, since FGW is a computationally prohibitive quadratic problem, we approximate it through anchor-based linearization. The resulting probabilistic transport plan provides a structurally consistent but noisy supervisory signal. Thus, we introduce a soft-target loss dynamically blending guidance from this plan with network predictions to build a learning framework robust to this noise. SoY achieves state-of-the-art performance on SPair-71k and AP-10k datasets, establishing a new benchmark in semantic correspondence without explicit geometric annotations. Code is available at Shape-of-You.