On Functional Dimension and Persistent Pseudodimension

TL;DR

This paper explores local complexity measures for ReLU neural networks, specifically local functional dimension and persistent pseudodimension, to better understand their redundancy and generalization behavior.

Contribution

It introduces and compares two local complexity measures for ReLU networks, providing insights into their relationship and implications for generalization and double descent.

Findings

Local functional dimension is easy to compute on finite data.

Persistent pseudodimension offers potential local generalization bounds.

The relationship between these measures informs understanding of model redundancy.

Abstract

For any fixed feedforward ReLU neural network architecture, it is well-known that many different parameter settings can determine the same function. It is less well-known that the degree of this redundancy is inhomogeneous across parameter space. In this work, we discuss two locally applicable complexity measures for ReLU network classes and what we know about the relationship between them: (1) the local functional dimension [14, 18], and (2) a local version of VC dimension that we call persistent pseudodimension. The former is easy to compute on finite batches of points; the latter should give local bounds on the generalization gap, which would inform an understanding of the mechanics of the double descent phenomenon [7].