Gromov Wasserstein Optimal Transport for Semantic Correspondences

TL;DR

This paper introduces a Gromov Wasserstein optimal transport method to improve semantic correspondence in images, achieving higher efficiency and competitive accuracy compared to existing ensemble-based approaches that combine features from large foundation models.

Contribution

The authors replace standard nearest neighbor matching with a Gromov Wasserstein optimal transport algorithm, significantly boosting performance and efficiency in semantic correspondence tasks.

Findings

Boosts DINOv2 baseline performance

Competitive with state-of-the-art methods using SD features

Achieves 5-10x efficiency improvement

Abstract

Establishing correspondences between image pairs is a long studied problem in computer vision. With recent large-scale foundation models showing strong zero-shot performance on downstream tasks including classification and segmentation, there has been interest in using the internal feature maps of these models for the semantic correspondence task. Recent works observe that features from DINOv2 and Stable Diffusion (SD) are complementary, the former producing accurate but sparse correspondences, while the latter produces spatially consistent correspondences. As a result, current state-of-the-art methods for semantic correspondence involve combining features from both models in an ensemble. While the performance of these methods is impressive, they are computationally expensive, requiring evaluating feature maps from large-scale foundation models. In this work we take a different approach, instead replacing SD features with a superior matching algorithm which is imbued with the desirable spatial consistency property. Specifically, we replace the standard nearest neighbours matching with an optimal transport algorithm that includes a Gromov Wasserstein spatial smoothness prior. We show that we can significantly boost the performance of the DINOv2 baseline, and be competitive and sometimes surpassing state-of-the-art methods using Stable Diffusion features, while being 5--10x more efficient. We make code available at https://github.com/fsnelgar/semantic_matching_gwot .