The training response law explains how deep neural networks learn

TL;DR

This paper introduces the training response law, a simple mathematical description of how deep neural networks learn, revealing the dynamics of the learning process and the evolution of network plasticity.

Contribution

It proposes the training response law that models neural tangent kernel behavior and explains the learning dynamics through a mean-field model.

Findings

Training response follows a power law decay multiplied by a response kernel.

Input space splits into sub-spaces during learning, driven by kernel competition.

Network plasticity decreases over time due to iterated splits and aging.

Abstract

Deep neural network is the widely applied technology in this decade. In spite of the fruitful applications, the mechanism behind that is still to be elucidated. We study the learning process with a very simple supervised learning encoding problem. As a result, we found a simple law, in the training response, which describes neural tangent kernel. The response consists of a power law like decay multiplied by a simple response kernel. We can construct a simple mean-field dynamical model with the law, which explains how the network learns. In the learning, the input space is split into sub-spaces along competition between the kernels. With the iterated splits and the aging, the network gets more complexity, but finally loses its plasticity.