A procedure for assessing of machine health index data prediction quality

TL;DR

This paper introduces a universal, statistically-based procedure for assessing the quality of machine health index data predictions, providing reliable binary classification of prognosis accuracy with potential for extension.

Contribution

The paper presents a novel, flexible procedure with multiple variants and criteria for evaluating machine health prediction quality, incorporating statistical thresholds for reliability assessment.

Findings

The procedure effectively distinguishes reliable from non-reliable predictions.

Application to literature-sourced data demonstrates method's versatility across degradation types.

Binary classification accuracy supports practical prognosis evaluation.

Abstract

The paper discusses the challenge of evaluating the prognosis quality of machine health index (HI) data. Many existing solutions in machine health forecasting involve visually assessing the quality of predictions to roughly gauge the similarity between predicted and actual samples, lacking precise measures or decisions. In this paper, we introduce a universal procedure with multiple variants and criteria. The overarching concept involves comparing predicted data with true HI time series, but each procedure variant has a specific pattern determined through statistical analysis. Additionally, a statistically established threshold is employed to classify the result as either a reliable or non-reliable prognosis. The criteria include both simple measures (MSE, MAPE) and more advanced ones (Space quantiles-inclusion factor, Kupiec's POF, and TUFF statistics). Depending on the criterion chosen, the pattern and decision-making process vary. To illustrate effectiveness, we apply the proposed procedure to HI data sourced from the literature, covering both warning (linear degradation) and critical (exponential degradation) stages. While the method yields a binary output, there is potential for extension to a multi-class classification. Furthermore, experienced users can use the quality measure expressed in percentage for more in-depth analysis.