A Light Touch Approach to Teaching Transformers Multi-view Geometry

TL;DR

This paper introduces a 'light touch' method guiding visual Transformers with epipolar lines to improve multi-view geometry understanding, enhancing object retrieval without requiring camera pose info at test-time.

Contribution

It proposes a novel approach that guides Transformers using epipolar lines, allowing flexible learning of geometry while maintaining geometric constraints, without needing pose data during testing.

Findings

Outperforms state-of-the-art object retrieval methods

Does not require camera pose information at test-time

Improves pose-invariant retrieval accuracy

Abstract

Transformers are powerful visual learners, in large part due to their conspicuous lack of manually-specified priors. This flexibility can be problematic in tasks that involve multiple-view geometry, due to the near-infinite possible variations in 3D shapes and viewpoints (requiring flexibility), and the precise nature of projective geometry (obeying rigid laws). To resolve this conundrum, we propose a "light touch" approach, guiding visual Transformers to learn multiple-view geometry but allowing them to break free when needed. We achieve this by using epipolar lines to guide the Transformer's cross-attention maps, penalizing attention values outside the epipolar lines and encouraging higher attention along these lines since they contain geometrically plausible matches. Unlike previous methods, our proposal does not require any camera pose information at test-time. We focus on pose-invariant object instance retrieval, where standard Transformer networks struggle, due to the large differences in viewpoint between query and retrieved images. Experimentally, our method outperforms state-of-the-art approaches at object retrieval, without needing pose information at test-time.