RoboNurse-VLA: Robotic Scrub Nurse System based on Vision-Language-Action Model

TL;DR

RoboNurse-VLA is an advanced robotic scrub nurse system that uses vision-language-action models to accurately recognize, grasp, and hand over surgical instruments in real-time, improving surgical efficiency and safety.

Contribution

This work introduces RoboNurse-VLA, a novel system integrating SAM 2 and Llama 2 models for precise, real-time instrument handling based on voice commands in surgical settings.

Findings

Achieves high success rates in instrument handover tasks.

Performs well with unseen and complex surgical tools.

Outperforms existing robotic nurse systems in accuracy and reliability.

Abstract

In modern healthcare, the demand for autonomous robotic assistants has grown significantly, particularly in the operating room, where surgical tasks require precision and reliability. Robotic scrub nurses have emerged as a promising solution to improve efficiency and reduce human error during surgery. However, challenges remain in terms of accurately grasping and handing over surgical instruments, especially when dealing with complex or difficult objects in dynamic environments. In this work, we introduce a novel robotic scrub nurse system, RoboNurse-VLA, built on a Vision-Language-Action (VLA) model by integrating the Segment Anything Model 2 (SAM 2) and the Llama 2 language model. The proposed RoboNurse-VLA system enables highly precise grasping and handover of surgical instruments in real-time based on voice commands from the surgeon. Leveraging state-of-the-art vision and language models, the system can address key challenges for object detection, pose optimization, and the handling of complex and difficult-to-grasp instruments. Through extensive evaluations, RoboNurse-VLA demonstrates superior performance compared to existing models, achieving high success rates in surgical instrument handovers, even with unseen tools and challenging items. This work presents a significant step forward in autonomous surgical assistance, showcasing the potential of integrating VLA models for real-world medical applications. More details can be found at https://robonurse-vla.github.io.