Statistical Uncertainty Quantification for Aggregate Performance Metrics in Machine Learning Benchmarks

TL;DR

This paper introduces statistical methods like bootstrapping and Bayesian modeling to quantify uncertainty in aggregate performance metrics across multiple machine learning tasks, providing more realistic evaluations.

Contribution

It applies and demonstrates the effectiveness of statistical uncertainty quantification techniques for aggregate ML benchmark metrics, enhancing evaluation accuracy.

Findings

Uncertainty quantification reveals model dominance in specific tasks.

Bayesian models provide credible intervals for performance metrics.

Visualization of task weightings helps interpret model performance.

Abstract

Modern artificial intelligence is supported by machine learning models (e.g., foundation models) that are pretrained on a massive data corpus and then adapted to solve a variety of downstream tasks. To summarize performance across multiple tasks, evaluation metrics are often aggregated into a summary metric, e.g., average accuracy across 10 question-answering tasks. When aggregating evaluation metrics, it is useful to incorporate uncertainty in the aggregate metric in order to gain a more realistic understanding of model performance. Our objective in this work is to demonstrate how statistical methodology can be used for quantifying uncertainty in metrics that have been aggregated across multiple tasks. The methods we emphasize are bootstrapping, Bayesian hierarchical (i.e., multilevel) modeling, and the visualization of task weightings that consider standard errors. These techniques reveal insights such as the dominance of a specific model for certain types of tasks despite an overall poor performance. We use a popular ML benchmark, the Visual Task Adaptation Benchmark (VTAB), to demonstrate the usefulness of our approaches.