Enumerating the k-fold configurations in multi-class classification problems

TL;DR

This paper discusses the enumeration of all k-fold configurations in multi-class classification to improve the reproducibility of performance evaluation, building on previous work that focused on binary classification.

Contribution

It introduces a new algorithm for enumerating k-fold configurations in multi-class classification, extending prior binary classification methods.

Findings

Proposed an algorithm for enumerating k-fold configurations in multi-class problems.

Enhanced the reproducibility of performance scores in cross-validation.

Addressed the irreproducibility crisis in AI performance evaluation.

Abstract

K-fold cross-validation is a widely used tool for assessing classifier performance. The reproducibility crisis faced by artificial intelligence partly results from the irreproducibility of reported k-fold cross-validation-based performance scores. Recently, we introduced numerical techniques to test the consistency of claimed performance scores and experimental setups. In a crucial use case, the method relies on the combinatorial enumeration of all k-fold configurations, for which we proposed an algorithm in the binary classification case.