Singular leaning coefficients and efficiency in learning theory

TL;DR

This paper investigates the theoretical learning efficiency of singular models like neural networks and mixture models by analyzing learning coefficients, extending results to models with ReLU and Softmax activations.

Contribution

It provides new theoretical insights into the learning coefficients of singular models, including deep linear, ReLU, and Softmax neural networks.

Findings

Learning coefficients quantify efficiency in singular models.

Results extend to ReLU and Softmax neural networks.

Theoretical analysis of learning models is advanced.

Abstract

Singular learning models with non-positive Fisher information matrices include neural networks, reduced-rank regression, Boltzmann machines, normal mixture models, and others. These models have been widely used in the development of learning machines. However, theoretical analysis is still in its early stages. In this paper, we examine learning coefficients, which indicate the general learning efficiency of deep linear learning models and three-layer neural network models with ReLU units. Finally, we extend the results to include the case of the Softmax function.