CTRL-GS: Cascaded Temporal Residue Learning for 4D Gaussian Splatting

TL;DR

This paper introduces CTRL-GS, a novel 4D Gaussian Splatting extension for dynamic scenes that hierarchically decomposes scenes using residual learning, achieving state-of-the-art quality and real-time rendering especially in complex, highly variable scenes.

Contribution

It proposes a cascaded residual learning framework for 4D Gaussian Splatting, enabling flexible modeling of dynamic scenes with large movements and occlusions.

Findings

Achieves state-of-the-art visual quality in dynamic scene rendering.

Demonstrates real-time rendering capabilities.

Performs especially well on complex scenes with large motions and occlusions.

Abstract

Recently, Gaussian Splatting methods have emerged as a desirable substitute for prior Radiance Field methods for novel-view synthesis of scenes captured with multi-view images or videos. In this work, we propose a novel extension to 4D Gaussian Splatting for dynamic scenes. Drawing on ideas from residual learning, we hierarchically decompose the dynamic scene into a "video-segment-frame" structure, with segments dynamically adjusted by optical flow. Then, instead of directly predicting the time-dependent signals, we model the signal as the sum of video-constant values, segment-constant values, and frame-specific residuals, as inspired by the success of residual learning. This approach allows more flexible models that adapt to highly variable scenes. We demonstrate state-of-the-art visual quality and real-time rendering on several established datasets, with the greatest improvements on complex scenes with large movements, occlusions, and fine details, where current methods degrade most.