Learning Curves for Decision Making in Supervised Machine Learning: A Survey

TL;DR

This survey reviews various learning curve models in supervised machine learning, categorizing approaches based on decision-making scenarios, questions answered, and resource types, to aid in model selection and early stopping.

Contribution

It provides a comprehensive framework for classifying learning curve approaches, integrating existing models and guiding their application in decision-making tasks.

Findings

Classified learning curve models into a unified framework

Identified key decision-making scenarios in machine learning

Highlighted the use of learning curves in model selection and early stopping

Abstract

Learning curves are a concept from social sciences that has been adopted in the context of machine learning to assess the performance of a learning algorithm with respect to a certain resource, e.g., the number of training examples or the number of training iterations. Learning curves have important applications in several machine learning contexts, most notably in data acquisition, early stopping of model training, and model selection. For instance, learning curves can be used to model the performance of the combination of an algorithm and its hyperparameter configuration, providing insights into their potential suitability at an early stage and often expediting the algorithm selection process. Various learning curve models have been proposed to use learning curves for decision making. Some of these models answer the binary decision question of whether a given algorithm at a certain budget will outperform a certain reference performance, whereas more complex models predict the entire learning curve of an algorithm. We contribute a framework that categorises learning curve approaches using three criteria: the decision-making situation they address, the intrinsic learning curve question they answer and the type of resources they use. We survey papers from the literature and classify them into this framework.