Splat2Real: Novel-view Scaling for Physical AI with 3D Gaussian Splatting

TL;DR

This paper introduces Splat2Real, a novel-view scaling method for physical AI that improves robustness to viewpoint shifts using a curriculum-based view selection strategy and a digital twin supervision approach.

Contribution

It presents a new viewpoint scaling technique, CN-Coverage, with a curriculum and fallback, enhancing stability and robustness in monocular depth perception under viewpoint shifts.

Findings

CN-Coverage reduces worst-case regressions compared to baseline policies.

GOL-Gated CN-Coverage offers the best stability at medium-high budgets.

Method improves safety and progress trade-offs in embodied perception tasks.

Abstract

Physical AI faces viewpoint shift between training and deployment, and novel-view robustness is essential for monocular RGB-to-3D perception. We cast Real2Render2Real monocular depth pretraining as imitation-learning-style supervision from a digital twin oracle: a student depth network imitates expert metric depth/visibility rendered from a scene mesh, while 3DGS supplies scalable novel-view observations. We present Splat2Real, centered on novel-view scaling: performance depends more on which views are added than on raw view count. We introduce CN-Coverage, a coverage+novelty curriculum that greedily selects views by geometry gain and an extrapolation penalty, plus a quality-aware guardrail fallback for low-reliability teachers. Across 20 TUM RGB-D sequences with step-matched budgets (N=0 to 2000 additional rendered views, with N unique <= 500 and resampling for larger budgets), naive scaling is unstable; CN-Coverage mitigates worst-case regressions relative to Robot/Coverage policies, and GOL-Gated CN-Coverage provides the strongest medium-high-budget stability with the lowest high-novelty tail error. Downstream control-proxy results versus N provides embodied-relevance evidence by shifting safety/progress trade-offs under viewpoint shift.