Real-to-Sim Registration of Deformable Soft Tissue with Position-Based Dynamics for Surgical Robot Autonomy

TL;DR

This paper introduces a real-to-sim registration method that combines vision-based perception with position-based dynamics to improve soft tissue modeling for autonomous surgical robots, enhancing safety and accuracy.

Contribution

It presents an online, continuous registration approach that integrates 3D perception with dynamic simulation, a novel method for deformable tissue modeling in robotic surgery.

Findings

Reduces registration error online during tissue manipulation

Achieves higher accuracy in occluded areas compared to fusion-based methods

Enhances safety and control accuracy in autonomous surgical procedures

Abstract

Autonomy in robotic surgery is very challenging in unstructured environments, especially when interacting with deformable soft tissues. The main difficulty is to generate model-based control methods that account for deformation dynamics during tissue manipulation. Previous works in vision-based perception can capture the geometric changes within the scene, however, model-based controllers integrated with dynamic properties, a more accurate and safe approach, has not been studied before. Considering the mechanic coupling between the robot and the environment, it is crucial to develop a registered, simulated dynamical model. In this work, we propose an online, continuous, real-to-sim registration method to bridge 3D visual perception with position-based dynamics (PBD) modeling of tissues. The PBD method is employed to simulate soft tissue dynamics as well as rigid tool interactions for model-based control. Meanwhile, a vision-based strategy is used to generate 3D reconstructed point cloud surfaces based on real-world manipulation, so as to register and update the simulation. To verify this real-to-sim approach, tissue experiments have been conducted on the da Vinci Research Kit. Our real-to-sim approach successfully reduces registration error online, which is especially important for safety during autonomous control. Moreover, it achieves higher accuracy in occluded areas than fusion-based reconstruction.