Specialized Foundation Models for Intelligent Operating Rooms

TL;DR

This paper introduces ORQA, a multimodal foundation model for surgical understanding that unifies visual, auditory, and structured data, significantly improving performance over generalist models in complex OR environments.

Contribution

The paper presents ORQA, a novel multimodal foundation model tailored for surgical settings, with a question-answering framework and smaller variants for diverse deployment needs.

Findings

ORQA outperforms general vision-language models in surgical scene perception.

Benchmark results show ORQA's superior accuracy across multiple surgical tasks.

Smaller ORQA models maintain strong performance with reduced computational requirements.

Abstract

Surgical procedures unfold in complex environments demanding coordination between surgical teams, tools, imaging and increasingly, intelligent robotic systems. Ensuring safety and efficiency in ORs of the future requires intelligent systems, like surgical robots, smart instruments and digital copilots, capable of understanding complex activities and hazards of surgeries. Yet, existing computational approaches, lack the breadth, and generalization needed for comprehensive OR understanding. We introduce ORQA, a multimodal foundation model unifying visual, auditory, and structured data for holistic surgical understanding. ORQA's question-answering framework empowers diverse tasks, serving as an intelligence core for a broad spectrum of surgical technologies. We benchmark ORQA against generalist vision-language models, including ChatGPT and Gemini, and show that while they struggle to perceive surgical scenes, ORQA delivers substantially stronger, consistent performance. Recognizing the extensive range of deployment settings across clinical practice, we design, and release a family of smaller ORQA models tailored to different computational requirements. This work establishes a foundation for the next wave of intelligent surgical solutions, enabling surgical teams and medical technology providers to create smarter and safer operating rooms.