Upper bounds on the Natarajan dimensions of some function classes

TL;DR

This paper provides upper bounds on the Natarajan dimension for specific multi-class function classes, aiding in understanding their learnability and generalization capabilities.

Contribution

It introduces new upper bounds on the Natarajan dimension for decision trees, random forests, and neural networks with various activations.

Findings

Upper bounds for multi-class decision trees and random forests.

Upper bounds for multi-class neural networks with different activations.

Implications for multi-class PAC learnability.

Abstract

The Natarajan dimension is a fundamental tool for characterizing multi-class PAC learnability, generalizing the Vapnik-Chervonenkis (VC) dimension from binary to multi-class classification problems. This work establishes upper bounds on Natarajan dimensions for certain function classes, including (i) multi-class decision tree and random forests, and (ii) multi-class neural networks with binary, linear and ReLU activations. These results may be relevant for describing the performance of certain multi-class learning algorithms.