Risk Bounds on MDL Estimators for Linear Regression Models with Application to Simple ReLU Neural Networks

TL;DR

This paper establishes tight risk bounds for MDL estimators applied to simple two-layer ReLU neural networks, demonstrating that the redundancy is small and the risk bound is independent of the number of hidden layer parameters.

Contribution

The paper introduces a method to design two-stage codes for linear regression models and applies it to simple ReLU neural networks, deriving risk bounds independent of hidden layer size.

Findings

Risk bounds of order O(d^2 log n / n) for simple ReLU networks.

Redundancy of two-stage codes is small due to eigenvalue bias in Fisher information.

Risk bounds are independent of the number of hidden layer parameters m.

Abstract

To investigate the theoretical foundations of deep learning from the viewpoint of the minimum description length (MDL) principle, we analyse risk bounds of MDL estimators based on two-stage codes for simple two-layers neural networks (NNs) with ReLU activation. For that purpose, we propose a method to design two-stage codes for linear regression models and establish an upper bound on the risk of the corresponding MDL estimators based on the theory of MDL estimators originated by Barron and Cover (1991). Then, we apply this result to the simple two-layers NNs with ReLU activation which consist of $d$ nodes in the input layer, $m$ nodes in the hidden layer and one output node. Since the object of estimation is only the $m$ weights from the hidden layer to the output node in our setting, this is an example of linear regression models. As a result, we show that the redundancy of the obtained two-stage codes is small owing to the fact that the eigenvalue distribution of the Fisher information matrix of the NNs is strongly biased, which was shown by Takeishi et al. (2023) and has been refined in this paper. That is, we establish a tight upper bound on the risk of our MDL estimators. Note that our risk bound for the simple ReLU networks, of which the leading term is $O(d^2 \log n /n)$, is independent of the number of parameters $m$.