GRVS: a Generalizable and Recurrent Approach to Monocular Dynamic View Synthesis

TL;DR

This paper introduces GRVS, a novel recurrent model for monocular dynamic view synthesis that disentangles camera and scene motion, outperforming existing scene-specific and diffusion-based methods on complex datasets.

Contribution

The paper presents a generalizable, recurrent approach with efficient plane sweep utilization for dynamic view synthesis, enabling fine-grained control and improved geometric reconstruction.

Findings

Outperforms scene-specific Gaussian Splatting approaches.

Achieves better geometric detail reconstruction in dynamic scenes.

Handles complex, high-resolution sequences with diverse scene dynamics.

Abstract

Synthesizing novel views from monocular videos of dynamic scenes remains a challenging problem. Scene-specific methods that optimize 4D representations with explicit motion priors often break down in highly dynamic regions where multi-view information is hard to exploit. Diffusion-based approaches that integrate camera control into large pre-trained models can produce visually plausible videos but frequently suffer from geometric inconsistencies across both static and dynamic areas. Both families of methods also require substantial computational resources. Building on the success of generalizable models for static novel view synthesis, we adapt the framework to dynamic inputs and propose a new model with two key components: (1) a recurrent loop that enables unbounded and asynchronous mapping between input and target videos and (2) an efficient use of plane sweeps over dynamic inputs to disentangle camera and scene motion, and achieve fine-grained, six-degrees-of-freedom camera controls. We train and evaluate our model on the UCSD dataset and on Kubric-4D-dyn, a new monocular dynamic dataset featuring longer, higher resolution sequences with more complex scene dynamics than existing alternatives. Our model outperforms four Gaussian Splatting-based scene-specific approaches, as well as two diffusion-based approaches in reconstructing fine-grained geometric details across both static and dynamic regions.