Interpretable global minima of deep ReLU neural networks on sequentially separable data

TL;DR

This paper constructs explicit zero-loss neural network classifiers for sequentially separable data, revealing the structure of global minima in terms of cumulative parameters and recursive truncation maps.

Contribution

It provides a novel explicit construction of global minima for deep ReLU networks on specific separable data, with a parameterization that clarifies the network's structure.

Findings

Global minimizers can be explicitly described with Q(M+2) parameters.

Configurations include well-separated clusters and sequential linear separability.

The approach offers insights into the structure of optimal neural network classifiers.

Abstract

We explicitly construct zero loss neural network classifiers. We write the weight matrices and bias vectors in terms of cumulative parameters, which determine truncation maps acting recursively on input space. The configurations for the training data considered are (i) sufficiently small, well separated clusters corresponding to each class, and (ii) equivalence classes which are sequentially linearly separable. In the best case, for $Q$ classes of data in $\mathbb{R}^M$, global minimizers can be described with $Q(M+2)$ parameters.