Mono-STAR: Mono-camera Scene-level Tracking and Reconstruction

TL;DR

Mono-STAR is a real-time 3D reconstruction system using a monocular camera that integrates semantic fusion, fast motion tracking, and topology change handling within a unified framework, advancing scene understanding.

Contribution

It introduces a novel optimization framework with optical-flow constraints and a semantic-aware deformation graph for improved scene reconstruction.

Findings

Outperforms existing state-of-the-art methods

Handles fast motion and topology changes effectively

Supports semantic fusion in real-time

Abstract

We present Mono-STAR, the first real-time 3D reconstruction system that simultaneously supports semantic fusion, fast motion tracking, non-rigid object deformation, and topological change under a unified framework. The proposed system solves a new optimization problem incorporating optical-flow-based 2D constraints to deal with fast motion and a novel semantic-aware deformation graph (SAD-graph) for handling topology change. We test the proposed system under various challenging scenes and demonstrate that it significantly outperforms existing state-of-the-art methods.