IndustReal: A Dataset for Procedure Step Recognition Handling Execution Errors in Egocentric Videos in an Industrial-Like Setting

TL;DR

This paper introduces the IndustReal dataset and the novel task of procedure step recognition (PSR) for industrial videos, emphasizing success measurement and robustness to procedural errors, with extensive annotations and benchmarks.

Contribution

The paper presents the IndustReal dataset with procedural and execution errors, and defines the new PSR task to assess success and order in industrial procedures.

Findings

Dataset includes procedural and execution errors, especially in validation/test sets.

Provides annotations and benchmarks for action recognition and assembly state detection.

Code and models are publicly available for reproducibility.

Abstract

Although action recognition for procedural tasks has received notable attention, it has a fundamental flaw in that no measure of success for actions is provided. This limits the applicability of such systems especially within the industrial domain, since the outcome of procedural actions is often significantly more important than the mere execution. To address this limitation, we define the novel task of procedure step recognition (PSR), focusing on recognizing the correct completion and order of procedural steps. Alongside the new task, we also present the multi-modal IndustReal dataset. Unlike currently available datasets, IndustReal contains procedural errors (such as omissions) as well as execution errors. A significant part of these errors are exclusively present in the validation and test sets, making IndustReal suitable to evaluate robustness of algorithms to new, unseen mistakes. Additionally, to encourage reproducibility and allow for scalable approaches trained on synthetic data, the 3D models of all parts are publicly available. Annotations and benchmark performance are provided for action recognition and assembly state detection, as well as the new PSR task. IndustReal, along with the code and model weights, is available at: https://github.com/TimSchoonbeek/IndustReal .