Toward Force Estimation in Robot-Assisted Surgery using Deep Learning with Vision and Robot State

TL;DR

This paper develops a deep learning model that estimates interaction forces in robot-assisted surgery using vision and robot state data, aiming to improve force feedback and skill evaluation without direct force sensors.

Contribution

It introduces a neural network combining RGB images and robot state for force estimation, demonstrating improved accuracy and robustness over single-input models and physics-based baselines.

Findings

Vision-based networks are sensitive to viewpoint shifts.

State-only networks are robust to workspace changes.

Combined vision and state inputs yield the best accuracy and real-time performance.

Abstract

Knowledge of interaction forces during teleoperated robot-assisted surgery could be used to enable force feedback to human operators and evaluate tissue handling skill. However, direct force sensing at the end-effector is challenging because it requires biocompatible, sterilizable, and cost-effective sensors. Vision-based deep learning using convolutional neural networks is a promising approach for providing useful force estimates, though questions remain about generalization to new scenarios and real-time inference. We present a force estimation neural network that uses RGB images and robot state as inputs. Using a self-collected dataset, we compared the network to variants that included only a single input type, and evaluated how they generalized to new viewpoints, workspace positions, materials, and tools. We found that vision-based networks were sensitive to shifts in viewpoints, while state-only networks were robust to changes in workspace. The network with both state and vision inputs had the highest accuracy for an unseen tool, and was moderately robust to changes in viewpoints. Through feature removal studies, we found that using only position features produced better accuracy than using only force features as input. The network with both state and vision inputs outperformed a physics-based baseline model in accuracy. It showed comparable accuracy but faster computation times than a baseline recurrent neural network, making it better suited for real-time applications.