Tailored Uncertainty Estimation for Deep Learning Systems

TL;DR

This paper introduces a framework for selecting and validating uncertainty estimation methods in deep learning, aiming to improve reliability and meet regulatory requirements by tailoring uncertainty quantification to specific use cases.

Contribution

It proposes a structured approach to match uncertainty estimators with use case requirements and provides validation strategies to identify weaknesses, enhancing trustworthy deep learning systems.

Findings

Framework effectively guides uncertainty estimator selection

Validation strategies uncover structural weaknesses

Supports compliance with machine learning regulations

Abstract

Uncertainty estimation bears the potential to make deep learning (DL) systems more reliable. Standard techniques for uncertainty estimation, however, come along with specific combinations of strengths and weaknesses, e.g., with respect to estimation quality, generalization abilities and computational complexity. To actually harness the potential of uncertainty quantification, estimators are required whose properties closely match the requirements of a given use case. In this work, we propose a framework that, firstly, structures and shapes these requirements, secondly, guides the selection of a suitable uncertainty estimation method and, thirdly, provides strategies to validate this choice and to uncover structural weaknesses. By contributing tailored uncertainty estimation in this sense, our framework helps to foster trustworthy DL systems. Moreover, it anticipates prospective machine learning regulations that require, e.g., in the EU, evidences for the technical appropriateness of machine learning systems. Our framework provides such evidences for system components modeling uncertainty.