DefSLAM: Tracking and Mapping of Deforming Scenes from Monocular Sequences

TL;DR

DefSLAM is a novel real-time monocular SLAM system capable of tracking and mapping deforming scenes, combining Shape-from-Template and Non-Rigid Structure-from-Motion techniques for applications like medical endoscopy.

Contribution

It introduces the first real-time monocular SLAM method that handles scene deformations by integrating SfT and NRSfM, enabling accurate 3D reconstruction of deforming scenes.

Findings

Successfully processes deforming scenes in real-time

Produces accurate 3D models in medical endoscopy sequences

Operates effectively in both laboratory and clinical environments

Abstract

Monocular SLAM algorithms perform robustly when observing rigid scenes, however, they fail when the observed scene deforms, for example, in medical endoscopy applications. We present DefSLAM, the first monocular SLAM capable of operating in deforming scenes in real-time. Our approach intertwines Shape-from-Template (SfT) and Non-Rigid Structure-from-Motion (NRSfM) techniques to deal with the exploratory sequences typical of SLAM. A deformation tracking thread recovers the pose of the camera and the deformation of the observed map, at frame rate, by means of SfT processing a template that models the scene shape-at-rest. A deformation mapping thread runs in parallel with the tracking to update the template, at keyframe rate, by means of an isometric NRSfM processing a batch of full perspective keyframes. In our experiments, DefSLAM processes close-up sequences of deforming scenes, both in a laboratory controlled experiment and in medical endoscopy sequences, producing accurate 3D models of the scene with respect to the moving camera.