An Uncertainty-based Human-in-the-loop System for Industrial Tool Wear Analysis

TL;DR

This paper introduces an uncertainty-based human-in-the-loop system using Monte-Carlo dropout for industrial image segmentation, improving trust and performance by identifying failed predictions for manual review.

Contribution

It presents a novel approach combining uncertainty measures with human-in-the-loop systems to enhance transparency and accuracy in industrial image segmentation tasks.

Findings

Uncertainty correlates with prediction quality in segmentation tasks.

Regression models can predict prediction failure with high accuracy.

The system improves performance over random human review strategies.

Abstract

Convolutional neural networks have shown to achieve superior performance on image segmentation tasks. However, convolutional neural networks, operating as black-box systems, generally do not provide a reliable measure about the confidence of their decisions. This leads to various problems in industrial settings, amongst others, inadequate levels of trust from users in the model's outputs as well as a non-compliance with current policy guidelines (e.g., EU AI Strategy). To address these issues, we use uncertainty measures based on Monte-Carlo dropout in the context of a human-in-the-loop system to increase the system's transparency and performance. In particular, we demonstrate the benefits described above on a real-world multi-class image segmentation task of wear analysis in the machining industry. Following previous work, we show that the quality of a prediction correlates with the model's uncertainty. Additionally, we demonstrate that a multiple linear regression using the model's uncertainties as independent variables significantly explains the quality of a prediction (\(R^2=0.718\)). Within the uncertainty-based human-in-the-loop system, the multiple regression aims at identifying failed predictions on an image-level. The system utilizes a human expert to label these failed predictions manually. A simulation study demonstrates that the uncertainty-based human-in-the-loop system increases performance for different levels of human involvement in comparison to a random-based human-in-the-loop system. To ensure generalizability, we show that the presented approach achieves similar results on the publicly available Cityscapes dataset.