ROBUSfT: Robust Real-Time Shape-from-Template, a C++ Library

TL;DR

ROBUSfT is a C++ library for real-time 3D shape tracking of deforming objects from monocular video, offering robustness to occlusions and large deformations, with high speed and no training required.

Contribution

It introduces a ready-to-use, real-time capable shape-from-template pipeline that handles large deformations and occlusions without training, outperforming existing methods.

Findings

Capable of 30 fps processing speed.

Robust against partial occlusions and discontinuities.

Outperforms state-of-the-art in challenging datasets.

Abstract

Tracking the 3D shape of a deforming object using only monocular 2D vision is a challenging problem. This is because one should (i) infer the 3D shape from a 2D image, which is a severely underconstrained problem, and (ii) implement the whole solution pipeline in real-time. The pipeline typically requires feature detection and matching, mismatch filtering, 3D shape inference and feature tracking algorithms. We propose ROBUSfT, a conventional pipeline based on a template containing the object's rest shape, texturemap and deformation law. ROBUSfT is ready-to-use, wide-baseline, capable of handling large deformations, fast up to 30 fps, free of training, and robust against partial occlusions and discontinuity in video frames. It outperforms the state-of-the-art methods in challenging datasets. ROBUSfT is implemented as a publicly available C++ library and we provide a tutorial on how to use it in https://github.com/mrshetab/ROBUSfT