SurgIRL: Towards Life-Long Learning for Surgical Automation by Incremental Reinforcement Learning

TL;DR

SurgIRL introduces a lifelong learning framework for surgical automation that incrementally acquires and reuses skills across multiple tasks, enhancing efficiency and adaptability in robotic surgery.

Contribution

The paper presents SurgIRL, a novel incremental reinforcement learning approach with a knowledge set and attention network for surgical task learning and transfer.

Findings

Efficiently learns ten surgical tasks in simulation.

Successfully transfers policies from simulation to real robot.

Enables sequential learning and reuse of surgical skills.

Abstract

Surgical automation holds immense potential to improve the outcome and accessibility of surgery. Recent studies use reinforcement learning to learn policies that automate different surgical tasks. However, these policies are developed independently and are limited in their reusability when the task changes, making it more time-consuming when robots learn to solve multiple tasks. Inspired by how human surgeons build their expertise, we train surgical automation policies through Surgical Incremental Reinforcement Learning (SurgIRL). SurgIRL aims to (1) acquire new skills by referring to external policies (knowledge) and (2) accumulate and reuse these skills to solve multiple unseen tasks incrementally (incremental learning). Our SurgIRL framework includes three major components. We first define an expandable knowledge set containing heterogeneous policies that can be helpful for surgical tasks. Then, we propose Knowledge Inclusive Attention Network with mAximum Coverage Exploration (KIAN-ACE), which improves learning efficiency by maximizing the coverage of the knowledge set during the exploration process. Finally, we develop incremental learning pipelines based on KIAN-ACE to accumulate and reuse learned knowledge and solve multiple surgical tasks sequentially. Our simulation experiments show that KIAN-ACE efficiently learns to automate ten surgical tasks separately or incrementally. We also evaluate our learned policies on the da Vinci Research Kit (dVRK) and demonstrate successful sim-to-real transfers.