Uncertainty Estimation in Machine Learning

TL;DR

This paper discusses the challenges of estimating uncertainty in machine learning models, especially complex and nonlinear ones, and explores the use of non-parametric techniques and high-performance computing to address these issues.

Contribution

It highlights the difficulties in uncertainty estimation for complex models and suggests leveraging modern supercomputing resources and non-parametric methods to improve evaluation.

Findings

Uncertainty estimation is challenging in nonlinear, complex models.

Pre-trained models like GPT-3 exemplify large-scale machine learning.

High-performance computing enables more effective uncertainty evaluation.

Abstract

Most machine learning techniques are based upon statistical learning theory, often simplified for the sake of computing speed. This paper is focused on the uncertainty aspect of mathematical modeling in machine learning. Regression analysis is chosen to further investigate the evaluation aspect of uncertainty in model coefficients and, more importantly, in the output feature value predictions. A survey demonstrates major stages in the conventional least squares approach to the creation of the regression model, along with its uncertainty estimation. On the other hand, it is shown that in machine learning the model complexity and severe nonlinearity become serious obstacles to uncertainty evaluation. Furthermore, the process of machine model training demands high computing power, not available at the level of personal computers. This is why so-called pre-trained models are widely used in such areas of machine learning as natural language processing. The latest example of a pre-trained model is the Generative Pre-trained Transformer 3 with hundreds of billions of parameters and a half-terabyte training dataset. Similarly, mathematical models built from real data are growing in complexity which is accompanied by the growing amount of training data. However, when machine models and their predictions are used in decision-making, one needs to estimate uncertainty and evaluate accompanying risks. This problem could be resolved with non-parametric techniques at the expense of greater demand for computing power, which can be offered by modern supercomputers available, including those utilizing graphical and tensor processing units along with the conventional central processors.