An adaptive and fully automatic method for estimating the 3D position of bendable instruments using endoscopic images

TL;DR

This paper introduces an automatic method to accurately estimate the 3D position of bendable surgical instruments using monocular endoscopic images with colored markers, enabling improved control and motion analysis.

Contribution

It presents a novel adaptive approach that combines image segmentation, marker corner detection, and 3D pose estimation accounting for mechanical play, specifically for bendable instruments in endoscopy.

Findings

Achieved RMS error of around 2-3 mm in laboratory tests.

Successfully estimated 3D tip position during real in vivo motions.

Method outperforms standard approaches in realistic conditions.

Abstract

Background. Flexible bendable instruments are key tools for performing surgical endoscopy. Being able to measure the 3D position of such instruments can be useful for various tasks, such as controlling automatically robotized instruments and analyzing motions. Methods. We propose an automatic method to infer the 3D pose of a single bending section instrument, using only the images provided by a monocular camera embedded at the tip of the endoscope. The proposed method relies on colored markers attached onto the bending section. The image of the instrument is segmented using a graph-based method and the corners of the markers are extracted by detecting the color transition along B{\'e}zier curves fitted on edge points. These features are accurately located and then used to estimate the 3D pose of the instrument using an adaptive model that allows to take into account the mechanical play between the instrument and its housing channel. Results. The feature extraction method provides good localization of markers corners with images of in vivo environment despite sensor saturation due to strong lighting. The RMS error on the estimation of the tip position of the instrument for laboratory experiments was 2.1, 1.96, 3.18 mm in the x, y and z directions respectively. Qualitative analysis in the case of in vivo images shows the ability to correctly estimate the 3D position of the instrument tip during real motions. Conclusions. The proposed method provides an automatic and accurate estimation of the 3D position of the tip of a bendable instrument in realistic conditions, where standard approaches fail.