SimpleMatch: A Simple and Strong Baseline for Semantic Correspondence

TL;DR

SimpleMatch is a lightweight, efficient framework for semantic correspondence that maintains high accuracy at low resolutions by addressing feature fusion issues and employing multi-scale supervision.

Contribution

It introduces a simple, effective method with a lightweight decoder and multi-scale loss to improve semantic correspondence at low resolutions, reducing memory usage significantly.

Findings

Achieves 84.1% PCK@0.1 on SPair-71k at 252x252 resolution

Reduces training memory by 51%

Performs better than state-of-the-art methods at lower resolutions

Abstract

Recent advances in semantic correspondence have been largely driven by the use of pre-trained large-scale models. However, a limitation of these approaches is their dependence on high-resolution input images to achieve optimal performance, which results in considerable computational overhead. In this work, we address a fundamental limitation in current methods: the irreversible fusion of adjacent keypoint features caused by deep downsampling operations. This issue is triggered when semantically distinct keypoints fall within the same downsampled receptive field (e.g., 16x16 patches). To address this issue, we present SimpleMatch, a simple yet effective framework for semantic correspondence that delivers strong performance even at low resolutions. We propose a lightweight upsample decoder that progressively recovers spatial detail by upsampling deep features to 1/4 resolution, and a multi-scale supervised loss that ensures the upsampled features retain discriminative features across different spatial scales. In addition, we introduce sparse matching and window-based localization to optimize training memory usage and reduce it by 51%. At a resolution of 252x252 (3.3x smaller than current SOTA methods), SimpleMatch achieves superior performance with 84.1% PCK@0.1 on the SPair-71k benchmark. We believe this framework provides a practical and efficient baseline for future research in semantic correspondence. Code is available at: https://github.com/hailong23-jin/SimpleMatch.