Towards Reducing Data Acquisition and Labeling for Defect Detection using Simulated Data

TL;DR

This paper explores cost-effective defect detection in manufacturing by combining synthetic and real X-ray data, demonstrating that domain adaptation and mixed training strategies can reduce annotation costs while maintaining high detection accuracy.

Contribution

It introduces a cost-efficient training approach using synthetic and unlabeled real data for defect detection, addressing domain shift challenges in manufacturing applications.

Findings

Sim-2-real domain adaptation is more cost-efficient than fully supervised training.

Mixing synthetic and unlabeled real data yields comparable or better results with fewer labeled samples.

Future research should focus on cost-efficiency of training strategies in machine learning.

Abstract

In many manufacturing settings, annotating data for machine learning and computer vision is costly, but synthetic data can be generated at significantly lower cost. Substituting the real-world data with synthetic data is therefore appealing for many machine learning applications that require large amounts of training data. However, relying solely on synthetic data is frequently inadequate for effectively training models that perform well on real-world data, primarily due to domain shifts between the synthetic and real-world data. We discuss approaches for dealing with such a domain shift when detecting defects in X-ray scans of aluminium wheels. Using both simulated and real-world X-ray images, we train an object detection model with different strategies to identify the training approach that generates the best detection results while minimising the demand for annotated real-world training samples. Our preliminary findings suggest that the sim-2-real domain adaptation approach is more cost-efficient than a fully supervised oracle - if the total number of available annotated samples is fixed. Given a certain number of labeled real-world samples, training on a mix of synthetic and unlabeled real-world data achieved comparable or even better detection results at significantly lower cost. We argue that future research into the cost-efficiency of different training strategies is important for a better understanding of how to allocate budget in applied machine learning projects.