A Novel Framework for Uncertainty Quantification via Proper Scores for Classification and Beyond

TL;DR

This thesis introduces a comprehensive framework for uncertainty quantification in machine learning based on proper scores, applicable across various tasks, with theoretical insights and practical evaluations including large language models and generative models.

Contribution

It develops a general bias-variance decomposition for proper scores, introduces new calibration error estimators, and applies kernel scores for evaluating generative models across domains.

Findings

Kernel score effectively evaluates generative models in multiple domains.

Proposed uncertainty estimation method outperforms existing baselines for large language models.

New calibration error estimators provide more accurate and interpretable calibration assessments.

Abstract

In this PhD thesis, we propose a novel framework for uncertainty quantification in machine learning, which is based on proper scores. Uncertainty quantification is an important cornerstone for trustworthy and reliable machine learning applications in practice. Usually, approaches to uncertainty quantification are problem-specific, and solutions and insights cannot be readily transferred from one task to another. Proper scores are loss functions minimized by predicting the target distribution. Due to their very general definition, proper scores apply to regression, classification, or even generative modeling tasks. We contribute several theoretical results, that connect epistemic uncertainty, aleatoric uncertainty, and model calibration with proper scores, resulting in a general and widely applicable framework. We achieve this by introducing a general bias-variance decomposition for strictly proper scores via functional Bregman divergences. Specifically, we use the kernel score, a kernel-based proper score, for evaluating sample-based generative models in various domains, like image, audio, and natural language generation. This includes a novel approach for uncertainty estimation of large language models, which outperforms state-of-the-art baselines. Further, we generalize the calibration-sharpness decomposition beyond classification, which motivates the definition of proper calibration errors. We then introduce a novel estimator for proper calibration errors in classification, and a novel risk-based approach to compare different estimators for squared calibration errors. Last, we offer a decomposition of the kernel spherical score, another kernel-based proper score, allowing a more fine-grained and interpretable evaluation of generative image models.