Who supervises the supervisor? Model monitoring in production using deep feature embeddings with applications to workpiece inspection

TL;DR

This paper introduces an unsupervised framework that monitors the performance of supervised classifiers in real-time during deployment by analyzing deep feature embeddings, addressing issues like data drift in manufacturing inspection systems.

Contribution

It proposes a novel method leveraging deep feature representations to detect data distribution changes without requiring labeled data, enhancing model monitoring in production environments.

Findings

Effective detection of data drift in real-time

Improved maintenance of classifier accuracy

Application to workpiece inspection processes

Abstract

The automation of condition monitoring and workpiece inspection plays an essential role in maintaining high quality as well as high throughput of the manufacturing process. To this end, the recent rise of developments in machine learning has lead to vast improvements in the area of autonomous process supervision. However, the more complex and powerful these models become, the less transparent and explainable they generally are as well. One of the main challenges is the monitoring of live deployments of these machine learning systems and raising alerts when encountering events that might impact model performance. In particular, supervised classifiers are typically build under the assumption of stationarity in the underlying data distribution. For example, a visual inspection system trained on a set of material surface defects generally does not adapt or even recognize gradual changes in the data distribution - an issue known as "data drift" - such as the emergence of new types of surface defects. This, in turn, may lead to detrimental mispredictions, e.g. samples from new defect classes being classified as non-defective. To this end, it is desirable to provide real-time tracking of a classifier's performance to inform about the putative onset of additional error classes and the necessity for manual intervention with respect to classifier re-training. Here, we propose an unsupervised framework that acts on top of a supervised classification system, thereby harnessing its internal deep feature representations as a proxy to track changes in the data distribution during deployment and, hence, to anticipate classifier performance degradation.