VisionNVS: Self-Supervised Inpainting for Novel View Synthesis under the Virtual-Shift Paradigm

TL;DR

VisionNVS introduces a self-supervised inpainting approach for novel view synthesis in autonomous driving, leveraging a Virtual-Shift paradigm and monocular depth proxies to improve geometric fidelity and visual quality without requiring ground truth for unseen views.

Contribution

It reformulates view synthesis as a self-supervised inpainting task using Virtual-Shift, eliminating the domain gap and enabling training solely on raw images.

Findings

Outperforms LiDAR-dependent baselines in geometric fidelity.

Achieves superior visual quality in novel view synthesis.

Provides a scalable solution for driving simulation.

Abstract

A fundamental bottleneck in Novel View Synthesis (NVS) for autonomous driving is the inherent supervision gap on novel trajectories: models are tasked with synthesizing unseen views during inference, yet lack ground truth images for these shifted poses during training. In this paper, we propose VisionNVS, a camera-only framework that fundamentally reformulates view synthesis from an ill-posed extrapolation problem into a self-supervised inpainting task. By introducing a ``Virtual-Shift'' strategy, we use monocular depth proxies to simulate occlusion patterns and map them onto the original view. This paradigm shift allows the use of raw, recorded images as pixel-perfect supervision, effectively eliminating the domain gap inherent in previous approaches. Furthermore, we address spatial consistency through a Pseudo-3D Seam Synthesis strategy, which integrates visual data from adjacent cameras during training to explicitly model real-world photometric discrepancies and calibration errors. Experiments demonstrate that VisionNVS achieves superior geometric fidelity and visual quality compared to LiDAR-dependent baselines, offering a robust solution for scalable driving simulation.