3D Moments from Near-Duplicate Photos

TL;DR

This paper introduces 3D Moments, a novel method to generate photorealistic 3D-interpolated videos from near-duplicate photos, capturing scene and camera motion with parallax.

Contribution

It presents a new scene representation using layered depth images with scene flow, enabling high-quality motion interpolation and camera control from just two photos.

Findings

Produces photorealistic 3D videos with parallax

Outperforms baseline methods on public datasets

Handles occlusions and scene dynamics effectively

Abstract

We introduce 3D Moments, a new computational photography effect. As input we take a pair of near-duplicate photos, i.e., photos of moving subjects from similar viewpoints, common in people's photo collections. As output, we produce a video that smoothly interpolates the scene motion from the first photo to the second, while also producing camera motion with parallax that gives a heightened sense of 3D. To achieve this effect, we represent the scene as a pair of feature-based layered depth images augmented with scene flow. This representation enables motion interpolation along with independent control of the camera viewpoint. Our system produces photorealistic space-time videos with motion parallax and scene dynamics, while plausibly recovering regions occluded in the original views. We conduct extensive experiments demonstrating superior performance over baselines on public datasets and in-the-wild photos. Project page: https://3d-moments.github.io/