Enhanced Velocity Field Modeling for Gaussian Video Reconstruction

TL;DR

This paper introduces FlowGaussian-VR, a novel velocity field modeling approach for Gaussian video reconstruction that improves visual quality and motion tracking in dynamic scenes by leveraging optical flow and adaptive densification.

Contribution

It proposes a flow-empowered velocity field scheme with a rendering pipeline and adaptive densification strategy, addressing overfitting and static scene limitations in prior methods.

Findings

Over 2.5 dB PSNR improvement in dynamic scenes

Reduced blurriness in dynamic textures

Enhanced Gaussian trajectory regularization and tracking

Abstract

High-fidelity 3D video reconstruction is essential for enabling real-time rendering of dynamic scenes with realistic motion in virtual and augmented reality (VR/AR). The deformation field paradigm of 3D Gaussian splatting has achieved near-photorealistic results in video reconstruction due to the great representation capability of deep deformation networks. However, in videos with complex motion and significant scale variations, deformation networks often overfit to irregular Gaussian trajectories, leading to suboptimal visual quality. Moreover, the gradient-based densification strategy designed for static scene reconstruction proves inadequate to address the absence of dynamic content. In light of these challenges, we propose a flow-empowered velocity field modeling scheme tailored for Gaussian video reconstruction, dubbed FlowGaussian-VR. It consists of two core components: a velocity field rendering (VFR) pipeline which enables optical flow-based optimization, and a flow-assisted adaptive densification (FAD) strategy that adjusts the number and size of Gaussians in dynamic regions. We validate our model's effectiveness on multi-view dynamic reconstruction and novel view synthesis with multiple real-world datasets containing challenging motion scenarios, demonstrating not only notable visual improvements (over 2.5 dB gain in PSNR) and less blurry artifacts in dynamic textures, but also regularized and trackable per-Gaussian trajectories.