Sample Complexity of Agnostic Multiclass Classification: Natarajan Dimension Strikes Back

TL;DR

This paper reveals that the sample complexity of agnostic multiclass classification depends on two dimensions, DS and Natarajan, providing nearly tight bounds and highlighting a fundamental difference from binary classification.

Contribution

It introduces a new bound involving both DS and Natarajan dimensions for agnostic multiclass PAC learning, showing the necessity of two parameters.

Findings

Sample complexity bounds involve DS and Natarajan dimensions.

The bounds are nearly tight up to logarithmic factors.

Multiclass learning inherently involves two structural parameters.

Abstract

The fundamental theorem of statistical learning states that binary PAC learning is governed by a single parameter -- the Vapnik-Chervonenkis (VC) dimension -- which determines both learnability and sample complexity. Extending this to multiclass classification has long been challenging, since Natarajan's work in the late 80s proposing the Natarajan dimension (Nat) as a natural analogue of VC. Daniely and Shalev-Shwartz (2014) introduced the DS dimension, later shown by Brukhim et al. (2022) to characterize multiclass learnability. Brukhim et al. also showed that Nat and DS can diverge arbitrarily, suggesting that multiclass learning is governed by DS rather than Nat. We show that agnostic multiclass PAC sample complexity is in fact governed by two distinct dimensions. Specifically, we prove nearly tight agnostic sample complexity bounds that, up to log factors, take the form $\frac{DS^{1.5}}{\epsilon} + \frac{Nat}{\epsilon^2}$ where $\epsilon$ is the excess risk. This bound is tight up to a $\sqrt{DS}$ factor in the first term, nearly matching known $Nat/\epsilon^2$ and $DS/\epsilon$ lower bounds. The first term reflects the DS-controlled regime, while the second shows that the Natarajan dimension still dictates asymptotic behavior for small $\epsilon$. Thus, unlike binary or online classification -- where a single dimension (VC or Littlestone) controls both phenomena -- multiclass learning inherently involves two structural parameters. Our technical approach departs from traditional agnostic learning methods based on uniform convergence or reductions to realizable cases. A key ingredient is a novel online procedure based on a self-adaptive multiplicative-weights algorithm performing a label-space reduction, which may be of independent interest.