Position: Why We Must Rethink Empirical Research in Machine Learning

TL;DR

This paper argues that empirical machine learning research is often overly confirmatory and needs to adopt more exploratory approaches to improve reliability and progress in the field.

Contribution

It highlights the epistemic limitations of current empirical methods and advocates for a shift towards more exploratory research practices in machine learning.

Findings

Current empirical ML research is predominantly confirmatory.

A lack of exploration leads to non-replicable results.

Adopting exploratory methods can enhance reliability.

Abstract

We warn against a common but incomplete understanding of empirical research in machine learning that leads to non-replicable results, makes findings unreliable, and threatens to undermine progress in the field. To overcome this alarming situation, we call for more awareness of the plurality of ways of gaining knowledge experimentally but also of some epistemic limitations. In particular, we argue most current empirical machine learning research is fashioned as confirmatory research while it should rather be considered exploratory.