Virtual Correspondence: Humans as a Cue for Extreme-View Geometry

TL;DR

This paper introduces virtual correspondences using human cues to improve camera pose estimation in extreme-view scenarios where traditional methods struggle due to lack of scene overlap.

Contribution

The paper proposes a novel concept of virtual correspondences based on human cues, enabling camera pose recovery without scene overlap, and integrates it with classic bundle adjustment.

Findings

Outperforms state-of-the-art in challenging extreme-view scenarios

Comparable to traditional methods in densely captured setups

Enhances downstream tasks like scene reconstruction and view synthesis

Abstract

Recovering the spatial layout of the cameras and the geometry of the scene from extreme-view images is a longstanding challenge in computer vision. Prevailing 3D reconstruction algorithms often adopt the image matching paradigm and presume that a portion of the scene is co-visible across images, yielding poor performance when there is little overlap among inputs. In contrast, humans can associate visible parts in one image to the corresponding invisible components in another image via prior knowledge of the shapes. Inspired by this fact, we present a novel concept called virtual correspondences (VCs). VCs are a pair of pixels from two images whose camera rays intersect in 3D. Similar to classic correspondences, VCs conform with epipolar geometry; unlike classic correspondences, VCs do not need to be co-visible across views. Therefore VCs can be established and exploited even if images do not overlap. We introduce a method to find virtual correspondences based on humans in the scene. We showcase how VCs can be seamlessly integrated with classic bundle adjustment to recover camera poses across extreme views. Experiments show that our method significantly outperforms state-of-the-art camera pose estimation methods in challenging scenarios and is comparable in the traditional densely captured setup. Our approach also unleashes the potential of multiple downstream tasks such as scene reconstruction from multi-view stereo and novel view synthesis in extreme-view scenarios.