On the Complexity of Labeled Datasets

TL;DR

This paper addresses the open problem of analytically computing the Shattering coefficient in Statistical Learning Theory, providing methods to estimate dataset complexity, sample sizes, and embedding effects for both binary and multi-class datasets.

Contribution

It introduces a novel approach using Topology, combinatorics, and data separability to compute the Shattering coefficient, advancing understanding of dataset complexity and sample requirements.

Findings

Computed Shattering coefficient for binary and multi-class datasets.

Estimated the number of hyperplanes needed in classification scenarios.

Determined training sample sizes for reliable supervised learning.

Abstract

The Statistical Learning Theory (SLT) provides the foundation to ensure that a supervised algorithm generalizes the mapping $f: \mathcal{X} \to \mathcal{Y}$ given $f$ is selected from its search space bias $\mathcal{F}$. SLT depends on the Shattering coefficient function $\mathcal{N}(\mathcal{F},n)$ to upper bound the empirical risk minimization principle, from which one can estimate the necessary training sample size to ensure the probabilistic learning convergence and, most importantly, the characterization of the capacity of $\mathcal{F}$, including its underfitting and overfitting abilities while addressing specific target problems. However, the analytical solution of the Shattering coefficient is still an open problem since the first studies by Vapnik and Chervonenkis in $1962$, which we address on specific datasets, in this paper, by employing equivalence relations from Topology, data separability results by Har-Peled and Jones, and combinatorics. Our approach computes the Shattering coefficient for both binary and multi-class datasets, leading to the following additional contributions: (i) the estimation of the required number of hyperplanes in the worst and best-case classification scenarios and the respective $\Omega$ and $O$ complexities; (ii) the estimation of the training sample sizes required to ensure supervised learning; and (iii) the comparison of dataset embeddings, once they (re)organize samples into some new space configuration. All results introduced and discussed along this paper are supported by the R package shattering (https://cran.r-project.org/web/packages/shattering).