GTT-Net: Learned Generalized Trajectory Triangulation

TL;DR

GTT-Net is a supervised learning framework that reconstructs sparse 3D dynamic geometry from multi-view data, outperforming existing methods in accuracy and robustness, and generalizing well across domains.

Contribution

It introduces a graph-theoretic learning approach for generalized trajectory triangulation, incorporating semantic motion priors and domain generalization capabilities.

Findings

Outperforms state-of-the-art in accuracy and robustness.

Learns and enforces semantic 3D motion priors.

Generalizes across different training and test domains.

Abstract

We present GTT-Net, a supervised learning framework for the reconstruction of sparse dynamic 3D geometry. We build on a graph-theoretic formulation of the generalized trajectory triangulation problem, where non-concurrent multi-view imaging geometry is known but global image sequencing is not provided. GTT-Net learns pairwise affinities modeling the spatio-temporal relationships among our input observations and leverages them to determine 3D geometry estimates. Experiments reconstructing 3D motion-capture sequences show GTT-Net outperforms the state of the art in terms of accuracy and robustness. Within the context of articulated motion reconstruction, our proposed architecture is 1) able to learn and enforce semantic 3D motion priors for shared training and test domains, while being 2) able to generalize its performance across different training and test domains. Moreover, GTT-Net provides a computationally streamlined framework for trajectory triangulation with applications to multi-instance reconstruction and event segmentation.