Real-time Rendering-based Surgical Instrument Tracking via Evolutionary Optimization

TL;DR

This paper introduces a real-time, rendering-based surgical instrument tracking method that uses evolutionary optimization to improve accuracy and efficiency in minimally invasive surgery, handling partial visibility and complex instrument articulation.

Contribution

It integrates CMA-ES with batch rendering for joint pose and configuration estimation, reducing inference time and enhancing robustness over prior methods.

Findings

Outperforms previous approaches in accuracy

Achieves faster inference times

Works effectively in joint angle-free and bi-manual settings

Abstract

Accurate and efficient tracking of surgical instruments is fundamental for Robot-Assisted Minimally Invasive Surgery. Although vision-based robot pose estimation has enabled markerless calibration without tedious physical setups, reliable tool tracking for surgical robots still remains challenging due to partial visibility and specialized articulation design of surgical instruments. Previous works in the field are usually prone to unreliable feature detections under degraded visual quality and data scarcity, whereas rendering-based methods often struggle with computational costs and suboptimal convergence. In this work, we incorporate CMA-ES, an evolutionary optimization strategy, into a versatile tracking pipeline that jointly estimates surgical instrument pose and joint configurations. Using batch rendering to efficiently evaluate multiple pose candidates in parallel, the method significantly reduces inference time and improves convergence robustness. The proposed framework further generalizes to joint angle-free and bi-manual tracking settings, making it suitable for both vision feedback control and online surgery video calibration. Extensive experiments on synthetic and real-world datasets demonstrate that the proposed method significantly outperforms prior approaches in both accuracy and runtime.