LeCoT: revisiting network architecture for two-view correspondence pruning

TL;DR

LeCoT introduces a novel transformer-based network for two-view correspondence pruning that effectively captures global context information without extra modules, outperforming existing methods across multiple vision tasks.

Contribution

We propose LeCoT, a new network architecture with a Spatial-Channel Fusion Transformer and a progressive prediction block for improved correspondence pruning.

Findings

LeCoT outperforms state-of-the-art methods in correspondence pruning.

LeCoT improves accuracy in relative pose and homography estimation.

LeCoT enhances visual localization and 3D reconstruction results.

Abstract

Two-view correspondence pruning aims to accurately remove incorrect correspondences (outliers) from initial ones and is widely applied to various computer vision tasks. Current popular strategies adopt multilayer perceptron (MLP) as the backbone, supplemented by additional modules to enhance the network ability to handle context information, which is a known limitation of MLPs. In contrast, we introduce a novel perspective for capturing correspondence context information without extra design modules. To this end, we design a two-view correspondence pruning network called LeCoT, which can naturally leverage global context information at different stages. Specifically, the core design of LeCoT is the Spatial-Channel Fusion Transformer block, a newly proposed component that efficiently utilizes both spatial and channel global context information among sparse correspondences. In addition, we integrate the proposed prediction block that utilizes correspondence features from intermediate stages to generate a probability set, which acts as guiding information for subsequent learning phases, allowing the network to more effectively capture robust global context information. Notably, this prediction block progressively refines the probability set, thereby mitigating the issue of information loss that is common in the traditional one. Extensive experiments prove that the proposed LeCoT outperforms state-of-the-art methods in correspondence pruning, relative pose estimation, homography estimation, visual localization, and $3$D~reconstruction tasks. The code is provided in https://github.com/Dailuanyuan2024/LeCoT-Revisiting-Network-Architecture-for-Two-View-Correspondence-Pruning.