# An augmented reality overlay for navigated prostatectomy using fiducial-free 2D–3D registration

## TL;DR

This paper introduces a markerless augmented reality system for prostatectomy using 2D-3D registration to help surgeons navigate during minimally invasive procedures.

## Contribution

The novel approach combines visual cues and registration techniques to enable real-time AR overlays in prostatectomies without fiducial markers.

## Key findings

- Differentiable rendering achieved a median target registration error of 6.11 mm on synthetic data.
- Both PnPRansac and differentiable rendering proved feasible for 2D–3D registration in prostatectomy navigation.
- A video-based markerless AR overlay was successfully demonstrated using in vivo and synthetic data.

## Abstract

Markerless navigation in minimally invasive surgery is still an unsolved challenge. Many proposed navigation systems for minimally invasive surgeries rely on stereoscopic images, while in clinical practice oftentimes monocular endoscopes are used. Combined with the lack of automatic video-based navigation systems for prostatectomies, this paper explores methods to tackle both research gaps at the same time for robot-assisted prostatectomies.

In order to realize a semi-automatic augmented reality overlay for navigated prostatectomy, the camera pose w.r.t. the prostate needs to be estimated. We developed a method where visual cues are drawn on top of the organ after an initial manual alignment, simultaneously creating matching landmarks on the 2D and 3D data. Starting from this key frame, the cues are then tracked in the endoscopic video. Both PnPRansac and differentiable rendering are then explored to perform 2D–3D registration for each frame.

We performed experiments on synthetic and in vivo data. On synthetic data differentiable rendering can achieve a median target registration error of 6.11 mm. Both PnPRansac and differentiable rendering are feasible methods for 2D–3D registration.

We demonstrated a video-based markerless augmented reality overlay for navigated prostatectomy, using visual cues as an anchor.

The online version contains supplementary material available at 10.1007/s11548-025-03374-5.