Optimal multiclass overfitting by sequence reconstruction from Hamming queries

TL;DR

This paper characterizes the extent of overfitting in multiclass classification with multiple queries, providing tight bounds and efficient algorithms that match theoretical limits.

Contribution

It resolves an open problem by precisely characterizing overfitting bias in multiclass classification and offers algorithms that achieve these bounds.

Findings

Overfitting bias is tightly bounded by rac{\u221a{k/(mn)}}{k/n}

Algorithms match the theoretical upper bounds for overfitting bias

Multiclass classification shows more resistance to overfitting than binary classification

Abstract

A primary concern of excessive reuse of test datasets in machine learning is that it can lead to overfitting. Multiclass classification was recently shown to be more resistant to overfitting than binary classification. In an open problem of COLT 2019, Feldman, Frostig, and Hardt ask to characterize the dependence of the amount of overfitting bias with the number of classes $m$, the number of accuracy queries $k$, and the number of examples in the dataset $n$. We resolve this problem and determine the amount of overfitting possible in multi-class classification. We provide computationally efficient algorithms that achieve overfitting bias of $\tilde{\Theta}(\max\{\sqrt{{k}/{(mn)}}, k/n\})$, matching the known upper bounds.