SuperPoint-E: local features for 3D reconstruction via tracking adaptation in endoscopy

TL;DR

SuperPoint-E is a novel local feature extraction method tailored for endoscopy videos, significantly enhancing 3D reconstruction quality by improving feature detection, description, and matching through tracking adaptation supervision.

Contribution

We introduce SuperPoint-E, a new feature extractor with a tracking adaptation strategy that outperforms existing methods in endoscopy-based 3D reconstruction tasks.

Findings

Denser and longer 3D reconstructions in endoscopy videos

Higher detection precision and feature survivability

More discriminative descriptors reducing the need for guided matching

Abstract

In this work, we focus on boosting the feature extraction to improve the performance of Structure-from-Motion (SfM) in endoscopy videos. We present SuperPoint-E, a new local feature extraction method that, using our proposed Tracking Adaptation supervision strategy, significantly improves the quality of feature detection and description in endoscopy. Extensive experimentation on real endoscopy recordings studies our approach's most suitable configuration and evaluates SuperPoint-E feature quality. The comparison with other baselines also shows that our 3D reconstructions are denser and cover more and longer video segments because our detector fires more densely and our features are more likely to survive (i.e. higher detection precision). In addition, our descriptor is more discriminative, making the guided matching step almost redundant. The presented approach brings significant improvements in the 3D reconstructions obtained, via SfM on endoscopy videos, compared to the original SuperPoint and the gold standard SfM COLMAP pipeline.