Accurate 3D Reconstruction of Dynamic Scenes from Monocular Image Sequences with Severe Occlusions

TL;DR

This paper presents a novel variational framework for accurate 3D reconstruction of dynamic scenes from monocular sequences, effectively handling severe occlusions and improving upon existing NRSfM methods.

Contribution

It introduces a shape prior integrated into the optimization process, enhancing reconstruction accuracy under occlusions and inaccurate correspondences.

Findings

Significantly outperforms state-of-the-art methods on synthetic and real data.

Effectively handles severe occlusions with improved correspondence establishment.

Provides detailed implementation insights for heterogeneous platforms.

Abstract

The paper introduces an accurate solution to dense orthographic Non-Rigid Structure from Motion (NRSfM) in scenarios with severe occlusions or, likewise, inaccurate correspondences. We integrate a shape prior term into variational optimisation framework. It allows to penalize irregularities of the time-varying structure on the per-pixel level if correspondence quality indicator such as an occlusion tensor is available. We make a realistic assumption that several non-occluded views of the scene are sufficient to estimate an initial shape prior, though the entire observed scene may exhibit non-rigid deformations. Experiments on synthetic and real image data show that the proposed framework significantly outperforms state of the art methods for correspondence establishment in combination with the state of the art NRSfM methods. Together with the profound insights into optimisation methods, implementation details for heterogeneous platforms are provided.