C3Po: Cross-View Cross-Modality Correspondence by Pointmap Prediction

TL;DR

This paper introduces C3, a new large-scale dataset for cross-view, cross-modality correspondence between ground photos and floor plans, and proposes a method to improve correspondence prediction in challenging scenarios.

Contribution

The paper presents a novel dataset C3 with 90K paired images and floor plans, and demonstrates improved correspondence prediction by training models on this data.

Findings

State-of-the-art models perform poorly on cross-modal correspondence tasks.

Training on C3 dataset improves RMSE by 34%.

Predicted correspondences enable camera pose estimation.

Abstract

Geometric models like DUSt3R have shown great advances in understanding the geometry of a scene from pairs of photos. However, they fail when the inputs are from vastly different viewpoints (e.g., aerial vs. ground) or modalities (e.g., photos vs. abstract drawings) compared to what was observed during training. This paper addresses a challenging version of this problem: predicting correspondences between ground-level photos and floor plans. Current datasets for joint photo-floor plan reasoning are limited, either lacking in varying modalities (VIGOR) or lacking in correspondences (WAFFLE). To address these limitations, we introduce a new dataset, C3, created by first reconstructing a number of scenes in 3D from Internet photo collections via structure-from-motion, then manually registering the reconstructions to floor plans gathered from the Internet, from which we can derive correspondences between images and floor plans. C3 contains 90K paired floor plans and photos across 597 scenes with 153M pixel-level correspondences and 85K camera poses. We find that state-of-the-art correspondence models struggle on this task. By training on our new data, we can improve on the best performing method by 34% in RMSE. We also use the predicted correspondences to estimate camera poses and evaluate performance using recall metrics. Lastly, we identify open challenges in cross-modal geometric reasoning that our dataset aims to help address.