Workflow Augmentation of Video Data for Event Recognition with Time-Sensitive Neural Networks

TL;DR

This paper introduces a novel workflow augmentation method for video data that enhances event recognition, especially for rare events, by creating artificial videos that increase event alternation frequency and improve classification accuracy.

Contribution

The paper presents a new workflow augmentation technique that constructs artificial videos based on workflow graphs, improving event recognition in surgical videos with temporal correlations.

Findings

Event alternation frequency increased by 26% in augmented videos.

Achieved 3% higher classification accuracy than state-of-the-art.

Achieved 7.8% higher precision in event detection.

Abstract

Supervised training of neural networks requires large, diverse and well annotated data sets. In the medical field, this is often difficult to achieve due to constraints in time, expert knowledge and prevalence of an event. Artificial data augmentation can help to prevent overfitting and improve the detection of rare events as well as overall performance. However, most augmentation techniques use purely spatial transformations, which are not sufficient for video data with temporal correlations. In this paper, we present a novel methodology for workflow augmentation and demonstrate its benefit for event recognition in cataract surgery. The proposed approach increases the frequency of event alternation by creating artificial videos. The original video is split into event segments and a workflow graph is extracted from the original annotations. Finally, the segments are assembled into new videos based on the workflow graph. Compared to the original videos, the frequency of event alternation in the augmented cataract surgery videos increased by 26%. Further, a 3% higher classification accuracy and a 7.8% higher precision was achieved compared to a state-of-the-art approach. Our approach is particularly helpful to increase the occurrence of rare but important events and can be applied to a large variety of use cases.