Confidence-aware Adversarial Learning for Self-supervised Semantic Matching

TL;DR

This paper introduces CAMNet, a confidence-aware adversarial learning framework for self-supervised semantic matching that improves accuracy by estimating confidence maps and refining predictions through an end-to-end training process.

Contribution

The paper presents a novel confidence-aware semantic matching network that estimates confidence maps and refines predictions, integrating a new hybrid loss and adversarial training for improved accuracy.

Findings

Achieves top performance on two public benchmarks.

Introduces a confidence estimation and refinement strategy.

Develops an end-to-end self-supervised adversarial learning method.

Abstract

In this paper, we aim to address the challenging task of semantic matching where matching ambiguity is difficult to resolve even with learned deep features. We tackle this problem by taking into account the confidence in predictions and develop a novel refinement strategy to correct partial matching errors. Specifically, we introduce a Confidence-Aware Semantic Matching Network (CAMNet) which instantiates two key ideas of our approach. First, we propose to estimate a dense confidence map for a matching prediction through self-supervised learning. Second, based on the estimated confidence, we refine initial predictions by propagating reliable matching to the rest of locations on the image plane. In addition, we develop a new hybrid loss in which we integrate a semantic alignment loss with a confidence loss, and an adversarial loss that measures the quality of semantic correspondence. We are the first that exploit confidence during refinement to improve semantic matching accuracy and develop an end-to-end self-supervised adversarial learning procedure for the entire matching network. We evaluate our method on two public benchmarks, on which we achieve top performance over the prior state of the art. We will release our source code at https://github.com/ShuaiyiHuang/CAMNet.