CATs: Cost Aggregation Transformers for Visual Correspondence

TL;DR

This paper introduces Cost Aggregation Transformers (CATs), a novel neural network architecture that leverages self-attention mechanisms for improved dense correspondence matching between semantically similar images, especially under challenging variations.

Contribution

The paper presents a new cost aggregation network using self-attention, multi-level aggregation, and appearance affinity modeling to enhance matching accuracy under severe deformations.

Findings

CATs outperform recent methods in dense correspondence tasks.

Inclusion of self-attention improves robustness to geometric variations.

Extensive ablations validate the effectiveness of architectural choices.

Abstract

We propose a novel cost aggregation network, called Cost Aggregation Transformers (CATs), to find dense correspondences between semantically similar images with additional challenges posed by large intra-class appearance and geometric variations. Cost aggregation is a highly important process in matching tasks, which the matching accuracy depends on the quality of its output. Compared to hand-crafted or CNN-based methods addressing the cost aggregation, in that either lacks robustness to severe deformations or inherit the limitation of CNNs that fail to discriminate incorrect matches due to limited receptive fields, CATs explore global consensus among initial correlation map with the help of some architectural designs that allow us to fully leverage self-attention mechanism. Specifically, we include appearance affinity modeling to aid the cost aggregation process in order to disambiguate the noisy initial correlation maps and propose multi-level aggregation to efficiently capture different semantics from hierarchical feature representations. We then combine with swapping self-attention technique and residual connections not only to enforce consistent matching but also to ease the learning process, which we find that these result in an apparent performance boost. We conduct experiments to demonstrate the effectiveness of the proposed model over the latest methods and provide extensive ablation studies. Project page is available at : https://sunghwanhong.github.io/CATs/.