PAPR in Motion: Seamless Point-level 3D Scene Interpolation

TL;DR

This paper introduces a novel point-level 3D scene interpolation method called 'PAPR in Motion' that effectively handles large non-rigid scene changes to produce smooth, coherent interpolations and novel view renderings without supervision.

Contribution

It extends the Proximity Attention Point Rendering technique to achieve temporally consistent, smooth 3D scene interpolation during significant non-rigid transformations.

Findings

Outperforms existing neural renderers on dynamic scenes

Maintains geometric and appearance consistency during large scene changes

Produces visually coherent, smooth interpolations in geometry and appearance

Abstract

We propose the problem of point-level 3D scene interpolation, which aims to simultaneously reconstruct a 3D scene in two states from multiple views, synthesize smooth point-level interpolations between them, and render the scene from novel viewpoints, all without any supervision between the states. The primary challenge is on achieving a smooth transition between states that may involve significant and non-rigid changes. To address these challenges, we introduce "PAPR in Motion", a novel approach that builds upon the recent Proximity Attention Point Rendering (PAPR) technique, which can deform a point cloud to match a significantly different shape and render a visually coherent scene even after non-rigid deformations. Our approach is specifically designed to maintain the temporal consistency of the geometric structure by introducing various regularization techniques for PAPR. The result is a method that can effectively bridge large scene changes and produce visually coherent and temporally smooth interpolations in both geometry and appearance. Evaluation across diverse motion types demonstrates that "PAPR in Motion" outperforms the leading neural renderer for dynamic scenes. For more results and code, please visit our project website at https://niopeng.github.io/PAPR-in-Motion/ .