Nonlinear Weighted Directed Acyclic Graph and A Priori Estimates for Neural Networks

TL;DR

This paper introduces a novel graph-theoretical framework for neural networks, including ResNet and DenseNet, and extends error analysis to provide input dimension-independent a priori estimates of generalization error.

Contribution

It presents a new graph-based formulation of neural networks and extends prior error analysis to DenseNet, offering dimension-independent a priori estimates.

Findings

Graph-theoretical formulation of neural networks including DenseNet

Extension of error bounds to DenseNet with dimension-independent estimates

Error bounds depend only on prior information, not input dimension

Abstract

In an attempt to better understand structural benefits and generalization power of deep neural networks, we firstly present a novel graph theoretical formulation of neural network models, including fully connected, residual network (ResNet) and densely connected networks (DenseNet). Secondly, we extend the error analysis of the population risk for two layer network \cite{ew2019prioriTwo} and ResNet \cite{e2019prioriRes} to DenseNet, and show further that for neural networks satisfying certain mild conditions, similar estimates can be obtained. These estimates are a priori in nature since they depend sorely on the information prior to the training process, in particular, the bounds for the estimation errors are independent of the input dimension.