Temporal-MPI: Enabling Multi-Plane Images for Dynamic Scene Modelling via Temporal Basis Learning

TL;DR

This paper introduces Temporal-MPI, a novel representation that encodes dynamic scenes over time using learned temporal bases, enabling real-time, high-quality novel view synthesis of moving scenes.

Contribution

The paper presents a new Temporal-MPI method that efficiently models dynamic scenes with a compact representation, allowing real-time rendering and improved speed over existing methods.

Findings

Temporal-MPI achieves real-time rendering of dynamic scenes.

It is more compact and faster than previous dynamic scene modeling techniques.

High visual quality is maintained in synthesized novel views.

Abstract

Novel view synthesis of static scenes has achieved remarkable advancements in producing photo-realistic results. However, key challenges remain for immersive rendering of dynamic scenes. One of the seminal image-based rendering method, the multi-plane image (MPI), produces high novel-view synthesis quality for static scenes. But modelling dynamic contents by MPI is not studied. In this paper, we propose a novel Temporal-MPI representation which is able to encode the rich 3D and dynamic variation information throughout the entire video as compact temporal basis and coefficients jointly learned. Time-instance MPI for rendering can be generated efficiently using mini-seconds by linear combinations of temporal basis and coefficients from Temporal-MPI. Thus novel-views at arbitrary time-instance will be able to be rendered via Temporal-MPI in real-time with high visual quality. Our method is trained and evaluated on Nvidia Dynamic Scene Dataset. We show that our proposed Temporal- MPI is much faster and more compact compared with other state-of-the-art dynamic scene modelling methods.