EgoMAGIC- An Egocentric Video Field Medicine Dataset for Training Perception Algorithms

TL;DR

EgoMAGIC is a comprehensive egocentric medical video dataset designed to advance perception algorithms, featuring 3,355 videos of medical tasks, object detection models, and baseline results for action detection.

Contribution

The paper introduces the EgoMAGIC dataset, a new resource for medical AI research, along with baseline action detection results and a challenge to foster further exploration.

Findings

Best model achieved average mAP 0.526 in action detection.

40 YOLO models trained on 1.95 million labels for 124 objects.

Dataset supports multiple computer vision tasks beyond action detection.

Abstract

This paper introduces EgoMAGIC (Medical Assistance, Guidance, Instruction, and Correction), an egocentric medical activity dataset collected as part of DARPA's Perceptually-enabled Task Guidance (PTG) program. This dataset comprises 3,355 videos of 50 medical tasks, with at least 50 labeled videos per task. The primary objective of the PTG program was to develop virtual assistants integrated into augmented reality headsets to assist users in performing complex tasks. To encourage exploration and research using this dataset, the medical training data has been released along with an action detection challenge focused on eight medical tasks. The majority of the videos were recorded using a head-mounted stereo camera with integrated audio. From this dataset, 40 YOLO models were trained using 1.95 million labels to detect 124 medical objects, providing a robust starting point for developers working on medical AI applications. In addition to introducing the dataset, this paper presents baseline results on action detection for the eight selected medical tasks across three models, with the best-performing method achieving average mAP 0.526. Although this paper primarily addresses action detection as the benchmark, the EgoMAGIC dataset is equally suitable for action recognition, object identification and detection, error detection, and other challenging computer vision tasks. The dataset is accessible via zenodo.org (DOI: 10.5281/zenodo.19239154).