What can we Learn by Predicting Accuracy?

TL;DR

This paper uses a data-driven, experimental approach with symbolic regression to discover an accurate, explainable formula predicting classifier accuracy across many datasets, offering new insights into loss function design.

Contribution

It introduces a novel, experimental method to derive an accurate, interpretable formula for predicting accuracy, challenging traditional intuition-based loss function development.

Findings

Discovered a formula with 0.96 Pearson correlation to accuracy

The formula explains previous insights on loss design

Achieved high correlation across 260 datasets

Abstract

This paper seeks to answer the following question: \textit{"What can we learn by predicting accuracy?"}. Indeed, classification is one of the most popular tasks in machine learning, and many loss functions have been developed to maximize this non-differentiable objective function. Unlike past work on loss function design, which was guided mainly by intuition and theory before being validated by experimentation, here we propose to approach this problem in the opposite way: we seek to extract knowledge by experimentation. This data-driven approach is similar to that used in physics to discover general laws from data. We used a symbolic regression method to automatically find a mathematical expression highly correlated with a linear classifier's accuracy. The formula discovered on more than 260 datasets of embeddings has a Pearson's correlation of 0.96 and a $r^2$ of 0.93. More interestingly, this formula is highly explainable and confirms insights from various previous papers on loss design. We hope this work will open new perspectives in the search for new heuristics leading to a deeper understanding of machine learning theory.