3D Video Loops from Asynchronous Input

TL;DR

This paper introduces a novel 3D video representation called Multi-Tile Video (MTV) that enables real-time, photorealistic 3D looping of scenes from asynchronous multi-view videos, suitable for mobile devices.

Contribution

It proposes a sparse 3D video representation and a two-stage pipeline for constructing 3D loops from asynchronous inputs, advancing the state of 3D scene looping technology.

Findings

Successfully generates photorealistic 3D loops in real time.

Reduces memory usage with the MTV representation.

Works with completely asynchronous multi-view videos.

Abstract

Looping videos are short video clips that can be looped endlessly without visible seams or artifacts. They provide a very attractive way to capture the dynamism of natural scenes. Existing methods have been mostly limited to 2D representations. In this paper, we take a step forward and propose a practical solution that enables an immersive experience on dynamic 3D looping scenes. The key challenge is to consider the per-view looping conditions from asynchronous input while maintaining view consistency for the 3D representation. We propose a novel sparse 3D video representation, namely Multi-Tile Video (MTV), which not only provides a view-consistent prior, but also greatly reduces memory usage, making the optimization of a 4D volume tractable. Then, we introduce a two-stage pipeline to construct the 3D looping MTV from completely asynchronous multi-view videos with no time overlap. A novel looping loss based on video temporal retargeting algorithms is adopted during the optimization to loop the 3D scene. Experiments of our framework have shown promise in successfully generating and rendering photorealistic 3D looping videos in real time even on mobile devices. The code, dataset, and live demos are available in https://limacv.github.io/VideoLoop3D_web/.