Video Dataset for Surgical Phase, Keypoint, and Instrument Recognition in Laparoscopic Surgery (PhaKIR)

TL;DR

The PhaKIR dataset offers a comprehensive, multi-institutional collection of laparoscopic surgery videos with detailed annotations for phase recognition, instrument keypoints, and segmentation, facilitating advanced computer vision research in surgical workflow analysis.

Contribution

This paper introduces the first multi-center dataset with synchronized annotations for surgical phases, instrument keypoints, and segmentations, supporting temporal and multi-task learning in minimally invasive surgery.

Findings

Dataset includes 485,875 frames for phase recognition.

Provides 19,435 frames with instrument keypoints and segmentations.

Served as the basis for the EndoVis Challenge at MICCAI 2024.

Abstract

Robotic- and computer-assisted minimally invasive surgery (RAMIS) is increasingly relying on computer vision methods for reliable instrument recognition and surgical workflow understanding. Developing such systems often requires large, well-annotated datasets, but existing resources often address isolated tasks, neglect temporal dependencies, or lack multi-center variability. We present the Surgical Procedure Phase, Keypoint, and Instrument Recognition (PhaKIR) dataset, comprising eight complete laparoscopic cholecystectomy videos recorded at three medical centers. The dataset provides frame-level annotations for three interconnected tasks: surgical phase recognition (485,875 frames), instrument keypoint estimation (19,435 frames), and instrument instance segmentation (19,435 frames). PhaKIR is, to our knowledge, the first multi-institutional dataset to jointly provide phase labels, instrument pose information, and pixel-accurate instrument segmentations, while also enabling the exploitation of temporal context since full surgical procedure sequences are available. It served as the basis for the PhaKIR Challenge as part of the Endoscopic Vision (EndoVis) Challenge at MICCAI 2024 to benchmark methods in surgical scene understanding, thereby further validating the dataset's quality and relevance. The dataset is publicly available upon request via the Zenodo platform.