Surg-InvNeRF: Invertible NeRF for 3D tracking and reconstruction in surgical vision

TL;DR

Surg-InvNeRF introduces an invertible NeRF architecture for improved long-term 3D point tracking in surgical scenarios, outperforming existing methods in accuracy and precision by leveraging rendering-based supervision and bidirectional mapping.

Contribution

The paper presents a novel invertible NeRF architecture enabling both 2D and 3D tracking, with efficient inference strategies and integration of kinematic data for surgical vision applications.

Findings

Surpasses state-of-the-art in 2D point tracking by nearly 50% in average precision.

First TTO approach to outperform feed-forward methods in 3D point tracking.

Effective integration of kinematic data enhances tracking accuracy.

Abstract

We proposed a novel test-time optimisation (TTO) approach framed by a NeRF-based architecture for long-term 3D point tracking. Most current methods in point tracking struggle to obtain consistent motion or are limited to 2D motion. TTO approaches frame the solution for long-term tracking as optimising a function that aggregates correspondences from other specialised state-of-the-art methods. Unlike the state-of-the-art on TTO, we propose parametrising such a function with our new invertible Neural Radiance Field (InvNeRF) architecture to perform both 2D and 3D tracking in surgical scenarios. Our approach allows us to exploit the advantages of a rendering-based approach by supervising the reprojection of pixel correspondences. It adapts strategies from recent rendering-based methods to obtain a bidirectional deformable-canonical mapping, to efficiently handle a defined workspace, and to guide the rays' density. It also presents our multi-scale HexPlanes for fast inference and a new algorithm for efficient pixel sampling and convergence criteria. We present results in the STIR and SCARE datasets, for evaluating point tracking and testing the integration of kinematic data in our pipeline, respectively. In 2D point tracking, our approach surpasses the precision and accuracy of the TTO state-of-the-art methods by nearly 50% on average precision, while competing with other approaches. In 3D point tracking, this is the first TTO approach, surpassing feed-forward methods while incorporating the benefits of a deformable NeRF-based reconstruction.