Convergence rates of deep ReLU networks for multiclass classification

TL;DR

This paper analyzes how quickly deep ReLU networks learn accurate class probabilities in multiclass classification, especially when some class probabilities are near zero, providing convergence rates based on margin conditions.

Contribution

It introduces convergence rate analysis for deep ReLU networks in multiclass classification, accounting for near-zero class probabilities and margin conditions.

Findings

Derived convergence rates depending on margin conditions

Identified phenomena when class probabilities are close to zero

Provided theoretical insights into probability estimation accuracy

Abstract

For classification problems, trained deep neural networks return probabilities of class memberships. In this work we study convergence of the learned probabilities to the true conditional class probabilities. More specifically we consider sparse deep ReLU network reconstructions minimizing cross-entropy loss in the multiclass classification setup. Interesting phenomena occur when the class membership probabilities are close to zero. Convergence rates are derived that depend on the near-zero behaviour via a margin-type condition.