Surgical Scene Understanding in the Era of Foundation AI Models: A Comprehensive Review

TL;DR

This comprehensive review discusses how recent foundation AI models and deep learning techniques are transforming surgical scene understanding, highlighting advancements, challenges, and future directions for clinical integration.

Contribution

It provides an extensive survey of state-of-the-art ML and DL technologies, including foundation models, in improving surgical scene analysis and discusses their clinical implications.

Findings

Significant progress in segmentation, tracking, and phase recognition.

Challenges include data variability and computational demands.

Future research needed for seamless clinical integration.

Abstract

Recent advancements in machine learning (ML) and deep learning (DL), particularly through the introduction of Foundation Models (FMs), have significantly enhanced surgical scene understanding within minimally invasive surgery (MIS). This paper surveys the integration of state-of-the-art ML and DL technologies, including Convolutional Neural Networks (CNNs), Vision Transformers (ViTs), and Foundation Models like the Segment Anything Model (SAM), into surgical workflows. These technologies improve segmentation accuracy, instrument tracking, and phase recognition in surgical scene understanding. The paper explores the challenges these technologies face, such as data variability and computational demands, and discusses ethical considerations and integration hurdles in clinical settings. Highlighting the roles of FMs, we bridge the technological capabilities with clinical needs and outline future research directions to enhance the adaptability, efficiency, and ethical alignment of AI applications in surgery. Our findings suggest that substantial progress has been made; however, more focused efforts are required to achieve seamless integration of these technologies into clinical workflows, ensuring they complement surgical practice by enhancing precision, reducing risks, and optimizing patient outcomes.