Find the Assembly Mistakes: Error Segmentation for Industrial Applications

TL;DR

This paper introduces StateDiffNet, a novel method for localizing assembly errors in industrial settings by detecting differences between correct and test assembly states, trained on synthetic data and effective on real-world videos.

Contribution

It is the first approach to accurately localize assembly errors in real ego-centric videos using change detection trained solely on synthetic data.

Findings

Successfully localizes errors in real-world videos

Effective in detecting unseen error types during training

Provides insights into change detection mechanisms

Abstract

Recognizing errors in assembly and maintenance procedures is valuable for industrial applications, since it can increase worker efficiency and prevent unplanned down-time. Although assembly state recognition is gaining attention, none of the current works investigate assembly error localization. Therefore, we propose StateDiffNet, which localizes assembly errors based on detecting the differences between a (correct) intended assembly state and a test image from a similar viewpoint. StateDiffNet is trained on synthetically generated image pairs, providing full control over the type of meaningful change that should be detected. The proposed approach is the first to correctly localize assembly errors taken from real ego-centric video data for both states and error types that are never presented during training. Furthermore, the deployment of change detection to this industrial application provides valuable insights and considerations into the mechanisms of state-of-the-art change detection algorithms. The code and data generation pipeline are publicly available at: https://timschoonbeek.github.io/error_seg.