SurgiPose: Estimating Surgical Tool Kinematics from Monocular Video for Surgical Robot Learning

TL;DR

SurgiPose introduces a differentiable rendering method to estimate surgical tool kinematics from monocular videos, enabling large-scale robot learning without ground truth data, and demonstrating comparable policy performance to ground truth-based training.

Contribution

The paper presents a novel monocular pose estimation approach using differentiable rendering for surgical tools, facilitating autonomous surgical robot learning from online videos.

Findings

Estimated kinematics enable policies with success rates similar to ground truth data.

The method successfully infers tool trajectories and joint angles from monocular videos.

Demonstrates feasibility of large-scale autonomous surgical policy learning.

Abstract

Imitation learning (IL) has shown immense promise in enabling autonomous dexterous manipulation, including learning surgical tasks. To fully unlock the potential of IL for surgery, access to clinical datasets is needed, which unfortunately lack the kinematic data required for current IL approaches. A promising source of large-scale surgical demonstrations is monocular surgical videos available online, making monocular pose estimation a crucial step toward enabling large-scale robot learning. Toward this end, we propose SurgiPose, a differentiable rendering based approach to estimate kinematic information from monocular surgical videos, eliminating the need for direct access to ground truth kinematics. Our method infers tool trajectories and joint angles by optimizing tool pose parameters to minimize the discrepancy between rendered and real images. To evaluate the effectiveness of our approach, we conduct experiments on two robotic surgical tasks: tissue lifting and needle pickup, using the da Vinci Research Kit Si (dVRK Si). We train imitation learning policies with both ground truth measured kinematics and estimated kinematics from video and compare their performance. Our results show that policies trained on estimated kinematics achieve comparable success rates to those trained on ground truth data, demonstrating the feasibility of using monocular video based kinematic estimation for surgical robot learning. By enabling kinematic estimation from monocular surgical videos, our work lays the foundation for large scale learning of autonomous surgical policies from online surgical data.