4D Visualization of Dynamic Events from Unconstrained Multi-View Videos

TL;DR

This paper introduces a data-driven, neural network-based method for 4D visualization of dynamic events from multi-view videos, enabling continuous navigation through space and time with editing capabilities.

Contribution

It presents a scene-specific self-supervised neural model that allows continuous 4D visualization and editing of multi-view videos captured by hand-held cameras.

Findings

Validated on challenging in-the-wild events with up to 15 cameras

Enables virtual camera movement through space and time

Allows editing and occlusion revealing in 4D visualization

Abstract

We present a data-driven approach for 4D space-time visualization of dynamic events from videos captured by hand-held multiple cameras. Key to our approach is the use of self-supervised neural networks specific to the scene to compose static and dynamic aspects of an event. Though captured from discrete viewpoints, this model enables us to move around the space-time of the event continuously. This model allows us to create virtual cameras that facilitate: (1) freezing the time and exploring views; (2) freezing a view and moving through time; and (3) simultaneously changing both time and view. We can also edit the videos and reveal occluded objects for a given view if it is visible in any of the other views. We validate our approach on challenging in-the-wild events captured using up to 15 mobile cameras.