Convergence of Deep Neural Networks with General Activation Functions and Pooling

TL;DR

This paper extends the mathematical understanding of deep neural network convergence by analyzing leaky ReLU and sigmoid activations, including pooling strategies, and establishes conditions for their uniform convergence as depth increases.

Contribution

It generalizes previous convergence results from ReLU to leaky ReLU and sigmoid activations, incorporating pooling effects and providing weaker conditions for sigmoid networks.

Findings

Sufficient conditions for convergence of leaky ReLU networks are confirmed.

Weaker convergence conditions are established for sigmoid activation functions.

Pooling strategies are incorporated into the convergence analysis.

Abstract

Deep neural networks, as a powerful system to represent high dimensional complex functions, play a key role in deep learning. Convergence of deep neural networks is a fundamental issue in building the mathematical foundation for deep learning. We investigated the convergence of deep ReLU networks and deep convolutional neural networks in two recent researches (arXiv:2107.12530, 2109.13542). Only the Rectified Linear Unit (ReLU) activation was studied therein, and the important pooling strategy was not considered. In this current work, we study the convergence of deep neural networks as the depth tends to infinity for two other important activation functions: the leaky ReLU and the sigmoid function. Pooling will also be studied. As a result, we prove that the sufficient condition established in arXiv:2107.12530, 2109.13542 is still sufficient for the leaky ReLU networks. For contractive activation functions such as the sigmoid function, we establish a weaker sufficient condition for uniform convergence of deep neural networks.