The kernel-balanced equation for deep neural networks

TL;DR

This paper introduces the kernel-balanced equation to explain the instability and scale mechanism in deep neural network training for dataset distribution estimation, highlighting how the scale decreases over time leading to instability.

Contribution

It derives a new kernel-balanced equation that phenomenologically describes the solution, instability, and scale mechanism in neural network training.

Findings

The estimation is unstable depending on data density and training duration.

The kernel-balanced equation explains the instability and the scale mechanism.

The network's output is a local average with a decreasing scale during training.

Abstract

Deep neural networks have shown many fruitful applications in this decade. A network can get the generalized function through training with a finite dataset. The degree of generalization is a realization of the proximity scale in the data space. Specifically, the scale is not clear if the dataset is complicated. Here we consider a network for the distribution estimation of the dataset. We show the estimation is unstable and the instability depends on the data density and training duration. We derive the kernel-balanced equation, which gives a short phenomenological description of the solution. The equation tells us the reason for the instability and the mechanism of the scale. The network outputs a local average of the dataset as a prediction and the scale of averaging is determined along the equation. The scale gradually decreases along training and finally results in instability in our case.