Upper Bounds for Local Learning Coefficients of Three-Layer Neural Networks

TL;DR

This paper derives an upper-bound formula for the local learning coefficient at singular points in three-layer neural networks, applicable to various activation functions, and clarifies its relation to known coefficients in one-dimensional input cases.

Contribution

It introduces a new upper-bound formula for the local learning coefficient at singular points in three-layer neural networks, extending applicability to a broader class of activation functions.

Findings

The upper bound matches known coefficients when input dimension is one.

The formula applies to general analytic activation functions, including swish and polynomial.

Provides a systematic understanding of how network parameters influence the learning coefficient.

Abstract

Three-layer neural networks are known to form singular learning models, and their Bayesian asymptotic behavior is governed by the learning coefficient, or real log canonical threshold. Although this quantity has been clarified for regular models and for some special singular models, broadly applicable methods for evaluating it in neural networks remain limited. Recently, a formula for the local learning coefficient of semiregular models was proposed, yielding an upper bound on the learning coefficient. However, this formula applies only to nonsingular points in the set of realization parameters and cannot be used at singular points. In particular, for three-layer neural networks, the resulting upper bound has been shown to differ substantially from learning coefficient values already known in some cases. In this paper, we derive an upper-bound formula for the local learning coefficient at singular points in three-layer neural networks. This formula can be interpreted as a counting rule under budget constraints and demand-supply constraints, and is applicable to general analytic activation functions. In particular, it covers the swish function and polynomial functions, extending previous results to a wider class of activation functions. We further show that, when the input dimension is one, the upper bound obtained here coincides with the already known learning coefficient, thereby partially resolving the discrepancy above. Our result also provides a systematic perspective on how the weight parameters of three-layer neural networks affect the learning coefficient.