Self-expressive Dictionary Learning for Dynamic 3D Reconstruction

TL;DR

This paper introduces a novel self-expressive dictionary learning framework for sparse 3D reconstruction of dynamic objects from multiple unsynchronized videos, effectively handling unknown temporal overlaps without sequencing information.

Contribution

It proposes a new compressed sensing approach that models 3D structures as a dictionary with self-expression, enabling reconstruction without explicit temporal sequencing.

Findings

Effective reconstruction on synthetic data

Successful application to real imagery

Outperforms existing methods in accuracy

Abstract

We target the problem of sparse 3D reconstruction of dynamic objects observed by multiple unsynchronized video cameras with unknown temporal overlap. To this end, we develop a framework to recover the unknown structure without sequencing information across video sequences. Our proposed compressed sensing framework poses the estimation of 3D structure as the problem of dictionary learning, where the dictionary is defined as an aggregation of the temporally varying 3D structures. Given the smooth motion of dynamic objects, we observe any element in the dictionary can be well approximated by a sparse linear combination of other elements in the same dictionary (i. e. self-expression). Moreover, the sparse coefficients describing a locally linear 3D structural interpolation reveal the local sequencing information. Our formulation optimizes a biconvex cost function that leverages a compressed sensing formulation and enforces both structural dependency coherence across video streams, as well as motion smoothness across estimates from common video sources. We further analyze the reconstructability of our approach under different capture scenarios, and its comparison and relation to existing methods. Experimental results on large amounts of synthetic data as well as real imagery demonstrate the effectiveness of our approach.