# Automatic and Real‐Time Surgeon's Gazing Point Detection From Surgical Videos Using Machine Learning and Mathematical Algorithm

## TL;DR

This paper introduces a system that automatically detects where a surgeon is looking during surgery using machine learning and math, which could help improve AI-assisted surgical tools.

## Contribution

A novel machine learning and mathematical algorithm system for real-time detection of a surgeon's gazing point in surgical videos.

## Key findings

- The system achieved 82.7% accuracy within a 216-pixel radius and 93.9% within a 324-pixel radius for pancreaticoduodenectomy.
- Time averaging improved accuracy, especially with a 5-second average.
- The system showed comparable performance in extended cholecystectomy and distal pancreatectomy.

## Abstract

Application of artificial intelligence (AI) in intraoperative imaging has been expanding rapidly. The surgeon's gazing point indicates the exact site of surgical procedures and concentrates critical information for AI applications. This study aimed to develop a machine learning‐based system to automatically detect the surgeon's gazing point from surgical video data.

Surgical instruments were detected using a deep‐learning model applied to images extracted from pancreaticoduodenectomy videos. Gazing points were estimated through a mathematical algorithm based on the axes and intersections of detected instruments, and time‐averaging was applied to enhance stability in real‐time analysis. After validation using pancreaticoduodenectomy cases, the system was subsequently applied to extended cholecystectomy and distal pancreatectomy cases to evaluate its applicability to other procedures.

Surgical instrument detection yielded AP50 of 60.5%. Gaze points detection achieved accuracies of 82.7% and 93.9% within 216‐ and 324‐pixel radii (9.42% and 21.2% of a 1440 × 1080 screen) in pancreaticoduodenectomy. When applied to extended cholecystectomy and pancreaticoduodenectomy distal pancreatectomy, our system demonstrated comparable performance, with an accuracy of 85.5% within the 324‐pixel radius. Time averaging improved accuracy, particularly with a 5‐s average.

Our system successfully detected the surgeon's gaze point across procedures, suggesting potential utility in future AI‐assisted surgery.

Sasaki and colleagues developed an automatic real‐time system to detect the surgeon's gazing point from surgical videos using a machine‐learning model and mathematical algorithm. Trained and validated on pancreaticoduodenectomy cases with high accuracy, the system showed comparable performance in distal pancreatectomy and extended cholecystectomy, highlighting its potential for future AI‐assisted surgery.