Information Scaling Law of Deep Neural Networks

TL;DR

This paper introduces an information scaling law for deep neural networks, specifically ConvACs, using information theory to interpret their internal organization and information flow.

Contribution

It proposes a novel information scaling law scheme that interprets DNNs' structure and information flow through rigorous information theory analysis.

Findings

Information entropy increases through ConvACs

Activation functions have an informational interpretation

The proposed law explains the network's internal organization

Abstract

With the rapid development of Deep Neural Networks (DNNs), various network models that show strong computing power and impressive expressive power are proposed. However, there is no comprehensive informational interpretation of DNNs from the perspective of information theory. Due to the nonlinear function and the uncertain number of layers and neural units used in the DNNs, the network structure shows nonlinearity and complexity. With the typical DNNs named Convolutional Arithmetic Circuits (ConvACs), the complex DNNs can be converted into mathematical formula. Thus, we can use rigorous mathematical theory especially the information theory to analyse the complicated DNNs. In this paper, we propose a novel information scaling law scheme that can interpret the network's inner organization by information theory. First, we show the informational interpretation of the activation function. Secondly, we prove that the information entropy increases when the information is transmitted through the ConvACs. Finally, we propose the information scaling law of ConvACs through making a reasonable assumption.