Branching Gaussian Processes with Applications to Spatiotemporal Reconstruction of 3D Trees

TL;DR

This paper introduces a novel branching Gaussian process prior for robustly reconstructing dynamic 3D tree structures from monocular images, effectively handling motion, camera error, and branching constraints.

Contribution

It presents a new probabilistic prior for modeling branching structures and an efficient inference scheme, enabling improved 3D reconstruction of dynamic trees from images.

Findings

Outperforms existing methods on a new plant dataset with motion

Enables principled comparison of models with different complexities

Proposes a new evaluation measure for branching 3D scene reconstruction

Abstract

We propose a robust method for estimating dynamic 3D curvilinear branching structure from monocular images. While 3D reconstruction from images has been widely studied, estimating thin structure has received less attention. This problem becomes more challenging in the presence of camera error, scene motion, and a constraint that curves are attached in a branching structure. We propose a new general-purpose prior, a branching Gaussian processes (BGP), that models spatial smoothness and temporal dynamics of curves while enforcing attachment between them. We apply this prior to fit 3D trees directly to image data, using an efficient scheme for approximate inference based on expectation propagation. The BGP prior's Gaussian form allows us to approximately marginalize over 3D trees with a given model structure, enabling principled comparison between tree models with varying complexity. We test our approach on a novel multi-view dataset depicting plants with known 3D structures and topologies undergoing small nonrigid motion. Our method outperforms a state-of-the-art 3D reconstruction method designed for non-moving thin structure. We evaluate under several common measures, and we propose a new measure for reconstructions of branching multi-part 3D scenes under motion.