Temporal View Synthesis of Dynamic Scenes through 3D Object Motion Estimation with Multi-Plane Images
Nagabhushan Somraj, Pranali Sancheti, Rajiv Soundararajan

TL;DR
This paper introduces a novel framework for temporal view synthesis of dynamic scenes by estimating 3D object motion using multi-plane images, enabling high-quality next-frame prediction in virtual reality with moving objects and users.
Contribution
The work presents a new method that decouples user and object motion, estimates 3D object displacement in MPI representation, and incorporates partial convolutions for better motion estimation.
Findings
Outperforms existing methods on synthetic and real datasets
Effectively handles object and user motion in dynamic scenes
Generates high-quality next-frame predictions with disocclusion filling
Abstract
The challenge of graphically rendering high frame-rate videos on low compute devices can be addressed through periodic prediction of future frames to enhance the user experience in virtual reality applications. This is studied through the problem of temporal view synthesis (TVS), where the goal is to predict the next frames of a video given the previous frames and the head poses of the previous and the next frames. In this work, we consider the TVS of dynamic scenes in which both the user and objects are moving. We design a framework that decouples the motion into user and object motion to effectively use the available user motion while predicting the next frames. We predict the motion of objects by isolating and estimating the 3D object motion in the past frames and then extrapolating it. We employ multi-plane images (MPI) as a 3D representation of the scenes and model the object…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Vision and Imaging · Computer Graphics and Visualization Techniques · Advanced Image Processing Techniques
