Characterization of Real-time Haptic Feedback from Multimodal Neural Network-based Force Estimates during Teleoperation

TL;DR

This study evaluates the real-time stability and transparency of neural network-based force feedback in surgical teleoperation, finding vision-only networks offer stable feedback, while state-based networks show instability during lateral manipulations.

Contribution

It provides the first real-time characterization of neural network force estimates in surgical teleoperation, comparing vision-only and state-based inputs for stability and transparency.

Findings

Vision-only networks provide stable force feedback.

State-based networks exhibit instability during lateral movements.

Neural network force estimates can be effectively used for real-time haptic feedback.

Abstract

Force estimation using neural networks is a promising approach to enable haptic feedback in minimally invasive surgical robots without end-effector force sensors. Various network architectures have been proposed, but none have been tested in real time with surgical-like manipulations. Thus, questions remain about the real-time transparency and stability of force feedback from neural network-based force estimates. We characterize the real-time impedance transparency and stability of force feedback rendered on a da Vinci Research Kit teleoperated surgical robot using neural networks with vision-only, state-only, and state and vision inputs. Networks were trained on an existing dataset of teleoperated manipulations without force feedback. To measure real-time stability and transparency during teleoperation with force feedback to the operator, we modeled a one-degree-of-freedom human and surgeon-side manipulandum that moved the patient-side robot to perform manipulations on silicone artificial tissue over various robot and camera configurations, and tools. We found that the networks using state inputs displayed more transparent impedance than a vision-only network. However, state-based networks displayed large instability when used to provide force feedback during lateral manipulation of the silicone. In contrast, the vision-only network showed consistent stability in all the evaluated directions. We confirmed the performance of the vision-only network for real-time force feedback in a demonstration with a human teleoperator.