Local Linear Recovery Guarantee of Deep Neural Networks at Overparameterization

TL;DR

This paper introduces the concept of local linear recovery (LLR) to analyze the sample efficiency of deep neural networks in overparameterized regimes, providing theoretical guarantees for function recovery.

Contribution

It defines LLR as a weaker form of recovery, establishes upper bounds on sample sizes needed for recovery, and proves these bounds are tight for two-layer tanh neural networks.

Findings

Functions expressible by narrower DNNs are recoverable from fewer samples.

Upper bounds on sample sizes for LLR are established and proven tight for certain networks.

The work provides a theoretical foundation for understanding DNN recovery in overparameterization.

Abstract

Determining whether deep neural network (DNN) models can reliably recover target functions at overparameterization is a critical yet complex issue in the theory of deep learning. To advance understanding in this area, we introduce a concept we term "local linear recovery" (LLR), a weaker form of target function recovery that renders the problem more amenable to theoretical analysis. In the sense of LLR, we prove that functions expressible by narrower DNNs are guaranteed to be recoverable from fewer samples than model parameters. Specifically, we establish upper limits on the optimistic sample sizes, defined as the smallest sample size necessary to guarantee LLR, for functions in the space of a given DNN. Furthermore, we prove that these upper bounds are achieved in the case of two-layer tanh neural networks. Our research lays a solid groundwork for future investigations into the recovery capabilities of DNNs in overparameterized scenarios.