No Calibration, No Depth, No Problem: Cross-Sensor View Synthesis with 3D Consistency

TL;DR

This paper introduces a scalable method for cross-sensor view synthesis that eliminates the need for calibration, enabling easier use of diverse RGB-X sensor data for 3D view generation.

Contribution

It proposes a match-densify-consolidate approach that removes calibration requirements, facilitating large-scale cross-sensor view synthesis without relying on 3D priors.

Findings

Improved view synthesis quality across different sensor modalities.

Reduced calibration effort in cross-sensor data collection.

Effective 3D consolidation using Gaussian Splatting.

Abstract

We present the first study of cross-sensor view synthesis across different modalities. We examine a practical, fundamental, yet widely overlooked problem: getting aligned RGB-X data, where most RGB-X prior work assumes such pairs exist and focuses on modality fusion, but it empirically requires huge engineering effort in calibration. We propose a match-densify-consolidate method. First, we perform RGB-X image matching followed by guided point densification. Using the proposed confidence-aware densification and self-matching filtering, we attain better view synthesis and later consolidate them in 3D Gaussian Splatting (3DGS). Our method uses no 3D priors for X-sensor and only assumes nearly no-cost COLMAP for RGB. We aim to remove the cumbersome calibration for various RGB-X sensors and advance the popularity of cross-sensor learning by a scalable solution that breaks through the bottleneck in large-scale real-world RGB-X data collection.