Machine Learning State-of-the-Art with Uncertainties

TL;DR

This paper demonstrates how incorporating confidence intervals in machine learning research, particularly in image classification, improves result communication and review processes, while discussing limitations and proposing workflow enhancements.

Contribution

It introduces the use of confidence intervals in ML result reporting, explores their benefits and limitations, and offers open-source workflows and suggestions for better research practices.

Findings

Confidence intervals enhance communication of accuracy results.

Reproducible workflows support transparent research.

Limitations of approximation methods are discussed.

Abstract

With the availability of data, hardware, software ecosystem and relevant skill sets, the machine learning community is undergoing a rapid development with new architectures and approaches appearing at high frequency every year. In this article, we conduct an exemplary image classification study in order to demonstrate how confidence intervals around accuracy measurements can greatly enhance the communication of research results as well as impact the reviewing process. In addition, we explore the hallmarks and limitations of this approximation. We discuss the relevance of this approach reflecting on a spotlight publication of ICLR22. A reproducible workflow is made available as an open-source adjoint to this publication. Based on our discussion, we make suggestions for improving the authoring and reviewing process of machine learning articles.