Enhancing Temporal Consistency in Video Editing by Reconstructing Videos with 3D Gaussian Splatting

TL;DR

This paper introduces Video-3DGS, a novel 3D Gaussian Splatting-based method that significantly improves temporal consistency and reconstruction quality in zero-shot video editing and reconstruction tasks.

Contribution

The paper presents a two-stage 3D Gaussian optimization process with MC-COLMAP for dynamic video reconstruction, enhancing temporal consistency in zero-shot video editing.

Findings

Improves video reconstruction PSNR by +3 and +7 over state-of-the-art methods.

Increases training efficiency by 1.9 to 4.5 times.

Ensures temporal consistency across 58 dynamic monocular videos.

Abstract

Recent advancements in zero-shot video diffusion models have shown promise for text-driven video editing, but challenges remain in achieving high temporal consistency. To address this, we introduce Video-3DGS, a 3D Gaussian Splatting (3DGS)-based video refiner designed to enhance temporal consistency in zero-shot video editors. Our approach utilizes a two-stage 3D Gaussian optimizing process tailored for editing dynamic monocular videos. In the first stage, Video-3DGS employs an improved version of COLMAP, referred to as MC-COLMAP, which processes original videos using a Masked and Clipped approach. For each video clip, MC-COLMAP generates the point clouds for dynamic foreground objects and complex backgrounds. These point clouds are utilized to initialize two sets of 3D Gaussians (Frg-3DGS and Bkg-3DGS) aiming to represent foreground and background views. Both foreground and background views are then merged with a 2D learnable parameter map to reconstruct full views. In the second stage, we leverage the reconstruction ability developed in the first stage to impose the temporal constraints on the video diffusion model. To demonstrate the efficacy of Video-3DGS on both stages, we conduct extensive experiments across two related tasks: Video Reconstruction and Video Editing. Video-3DGS trained with 3k iterations significantly improves video reconstruction quality (+3 PSNR, +7 PSNR increase) and training efficiency (x1.9, x4.5 times faster) over NeRF-based and 3DGS-based state-of-art methods on DAVIS dataset, respectively. Moreover, it enhances video editing by ensuring temporal consistency across 58 dynamic monocular videos.