FreeSim: Toward Free-viewpoint Camera Simulation in Driving Scenes

TL;DR

FreeSim introduces a novel camera simulation approach for autonomous driving that generates high-quality images from viewpoints beyond recorded trajectories, enabling better off-trajectory view synthesis despite data limitations.

Contribution

The paper presents a generative enhancement model and a progressive reconstruction strategy to improve off-trajectory viewpoint rendering in driving scene simulation.

Findings

Supports high-quality off-trajectory view synthesis over 3 meters deviation

Uses a generative model conditioned on degraded renderings

Employs progressive reconstruction for view expansion

Abstract

We propose FreeSim, a camera simulation method for autonomous driving. FreeSim emphasizes high-quality rendering from viewpoints beyond the recorded ego trajectories. In such viewpoints, previous methods have unacceptable degradation because the training data of these viewpoints is unavailable. To address such data scarcity, we first propose a generative enhancement model with a matched data construction strategy. The resulting model can generate high-quality images in a viewpoint slightly deviated from the recorded trajectories, conditioned on the degraded rendering of this viewpoint. We then propose a progressive reconstruction strategy, which progressively adds generated images of unrecorded views into the reconstruction process, starting from slightly off-trajectory viewpoints and moving progressively farther away. With this progressive generation-reconstruction pipeline, FreeSim supports high-quality off-trajectory view synthesis under large deviations of more than 3 meters.