Motion-from-Blur: 3D Shape and Motion Estimation of Motion-blurred Objects in Videos

TL;DR

This paper introduces a novel method to jointly estimate 3D shape, motion, and appearance of fast-moving, motion-blurred objects in videos using differentiable rendering and a generative model, outperforming previous approaches.

Contribution

It presents a comprehensive framework that models motion blur, shape, and motion parameters simultaneously, including abrupt changes like bounces, with sub-frame accuracy.

Findings

Outperforms previous methods in fast-moving object deblurring

Accurately estimates 3D shape and motion from blurred videos

Handles abrupt motion changes with piece-wise polynomial modeling

Abstract

We propose a method for jointly estimating the 3D motion, 3D shape, and appearance of highly motion-blurred objects from a video. To this end, we model the blurred appearance of a fast moving object in a generative fashion by parametrizing its 3D position, rotation, velocity, acceleration, bounces, shape, and texture over the duration of a predefined time window spanning multiple frames. Using differentiable rendering, we are able to estimate all parameters by minimizing the pixel-wise reprojection error to the input video via backpropagating through a rendering pipeline that accounts for motion blur by averaging the graphics output over short time intervals. For that purpose, we also estimate the camera exposure gap time within the same optimization. To account for abrupt motion changes like bounces, we model the motion trajectory as a piece-wise polynomial, and we are able to estimate the specific time of the bounce at sub-frame accuracy. Experiments on established benchmark datasets demonstrate that our method outperforms previous methods for fast moving object deblurring and 3D reconstruction.