Universal Correspondence Network

TL;DR

This paper introduces a deep learning framework that learns a feature space for accurate geometric and semantic visual correspondences, outperforming prior methods in speed and accuracy across various datasets.

Contribution

It proposes a fully convolutional architecture with a novel correspondence contrastive loss and a spatial transformer, enabling faster training, improved accuracy, and applicability to diverse matching tasks.

Findings

Outperforms prior methods on KITTI, PASCAL, and CUB-2011 datasets.

Achieves faster training and testing with $O(n)$ complexity.

Enhances semantic correspondence accuracy using a convolutional spatial transformer.

Abstract

We present a deep learning framework for accurate visual correspondences and demonstrate its effectiveness for both geometric and semantic matching, spanning across rigid motions to intra-class shape or appearance variations. In contrast to previous CNN-based approaches that optimize a surrogate patch similarity objective, we use deep metric learning to directly learn a feature space that preserves either geometric or semantic similarity. Our fully convolutional architecture, along with a novel correspondence contrastive loss allows faster training by effective reuse of computations, accurate gradient computation through the use of thousands of examples per image pair and faster testing with $O(n)$ feed forward passes for $n$ keypoints, instead of $O(n^2)$ for typical patch similarity methods. We propose a convolutional spatial transformer to mimic patch normalization in traditional features like SIFT, which is shown to dramatically boost accuracy for semantic correspondences across intra-class shape variations. Extensive experiments on KITTI, PASCAL, and CUB-2011 datasets demonstrate the significant advantages of our features over prior works that use either hand-constructed or learned features.