Trimming the Risk: Towards Reliable Continuous Training for Deep Learning Inspection Systems

TL;DR

This paper presents a robust continuous training approach for deep learning inspection systems that filters out unreliable data to prevent performance degradation, ensuring reliable model updates in industrial manufacturing contexts.

Contribution

It introduces a two-stage filtering process using confidence measures and variational auto-encoders to select trustworthy data for model fine-tuning, reducing the risk of silent performance decline.

Findings

Less than 9% of erroneous self-labeled data retained after filtering.

Model performance improved by up to 14% on production data.

Method effectively prevents catastrophic forgetting during continuous training.

Abstract

The industry increasingly relies on deep learning (DL) technology for manufacturing inspections, which are challenging to automate with rule-based machine vision algorithms. DL-powered inspection systems derive defect patterns from labeled images, combining human-like agility with the consistency of a computerized system. However, finite labeled datasets often fail to encompass all natural variations necessitating Continuous Training (CT) to regularly adjust their models with recent data. Effective CT requires fresh labeled samples from the original distribution; otherwise, selfgenerated labels can lead to silent performance degradation. To mitigate this risk, we develop a robust CT-based maintenance approach that updates DL models using reliable data selections through a two-stage filtering process. The initial stage filters out low-confidence predictions, as the model inherently discredits them. The second stage uses variational auto-encoders and histograms to generate image embeddings that capture latent and pixel characteristics, then rejects the inputs of substantially shifted embeddings as drifted data with erroneous overconfidence. Then, a fine-tuning of the original DL model is executed on the filtered inputs while validating on a mixture of recent production and original datasets. This strategy mitigates catastrophic forgetting and ensures the model adapts effectively to new operational conditions. Evaluations on industrial inspection systems for popsicle stick prints and glass bottles using critical real-world datasets showed less than 9% of erroneous self-labeled data are retained after filtering and used for fine-tuning, improving model performance on production data by up to 14% without compromising its results on original validation data.