4DSloMo: 4D Reconstruction for High Speed Scene with Asynchronous Capture

TL;DR

This paper introduces a novel 4D reconstruction system that uses asynchronous low FPS cameras and a generative diffusion model to achieve high-speed scene capture and artifact correction, surpassing traditional methods.

Contribution

The work presents a new asynchronous capture scheme combined with a diffusion-based artifact fix model for high-speed 4D scene reconstruction from low FPS cameras.

Findings

Achieves 100-200 FPS effective frame rate without high-speed cameras.

Significantly improves reconstruction quality over synchronous capture methods.

Effectively reduces artifacts and maintains temporal consistency in 4D reconstructions.

Abstract

Reconstructing fast-dynamic scenes from multi-view videos is crucial for high-speed motion analysis and realistic 4D reconstruction. However, the majority of 4D capture systems are limited to frame rates below 30 FPS (frames per second), and a direct 4D reconstruction of high-speed motion from low FPS input may lead to undesirable results. In this work, we propose a high-speed 4D capturing system only using low FPS cameras, through novel capturing and processing modules. On the capturing side, we propose an asynchronous capture scheme that increases the effective frame rate by staggering the start times of cameras. By grouping cameras and leveraging a base frame rate of 25 FPS, our method achieves an equivalent frame rate of 100-200 FPS without requiring specialized high-speed cameras. On processing side, we also propose a novel generative model to fix artifacts caused by 4D sparse-view reconstruction, as asynchrony reduces the number of viewpoints at each timestamp. Specifically, we propose to train a video-diffusion-based artifact-fix model for sparse 4D reconstruction, which refines missing details, maintains temporal consistency, and improves overall reconstruction quality. Experimental results demonstrate that our method significantly enhances high-speed 4D reconstruction compared to synchronous capture.