# An Automated Video Analysis System for Retrospective Assessment and Real-Time Monitoring of Endoscopic Procedures (with Video)

## TL;DR

This paper introduces EndoAdd, an automated system for analyzing endoscopic videos to improve quality control and real-time monitoring during procedures.

## Contribution

The novel contribution is the development of EndoAdd, a system combining YOLO-v5 and hidden Markov models for instrument detection and video analysis in endoscopy.

## Key findings

- EndoAdd achieved over 97% accuracy in identifying 10 endoscopic instruments on the test dataset.
- Heatmaps were successfully generated for both retrospective and real-time endoscopic procedure analysis.
- The system demonstrated high precision, recall, and F1-score metrics, with area under the curve values exceeding 0.94.

## Abstract

Background and Aims: Accurate recognition of endoscopic instruments facilitates quantitative evaluation and quality control of endoscopic procedures. However, no relevant research has been reported. In this study, we aimed to develop a computer-assisted system, EndoAdd, for automated endoscopic surgical video analysis based on our dataset of endoscopic instrument images. Methods: Large training and validation datasets containing 45,143 images of 10 different endoscopic instruments and a test dataset of 18,375 images collected from several medical centers were used in this research. Annotated image frames were used to train the state-of-the-art object detection model, YOLO-v5, to identify the instruments. Based on the frame-level prediction results, we further developed a hidden Markov model to perform video analysis and generate heatmaps to summarize the videos. Results: EndoAdd achieved high accuracy (>97%) on the test dataset for all 10 endoscopic instrument types. The mean average accuracy, precision, recall, and F1-score were 99.1%, 92.0%, 88.8%, and 89.3%, respectively. The area under the curve values exceeded 0.94 for all instrument types. Heatmaps of endoscopic procedures were generated for both retrospective and real-time analyses. Conclusions: We successfully developed an automated endoscopic video analysis system, EndoAdd, which supports retrospective assessment and real-time monitoring. It can be used for data analysis and quality control of endoscopic procedures in clinical practice.