Analytical Solution of a Three-layer Network with a Matrix Exponential Activation Function

TL;DR

This paper derives an analytical solution for a three-layer neural network with a matrix exponential activation, highlighting the theoretical advantages of depth and non-linear activations over shallow networks.

Contribution

It provides the first analytical solution for a three-layer network with matrix exponential activation, demonstrating the theoretical power of depth and non-linearity.

Findings

Three-layer network with matrix exponential activation has analytical solutions.

Depth and non-linear activation enhance the network's solving power.

One-layer networks can only solve linear equations.

Abstract

In practice, deeper networks tend to be more powerful than shallow ones, but this has not been understood theoretically. In this paper, we find the analytical solution of a three-layer network with a matrix exponential activation function, i.e., $$ f(X)=W_3\exp(W_2\exp(W_1X)), X\in \mathbb{C}^{d\times d} $$ have analytical solutions for the equations $$ Y_1=f(X_1),Y_2=f(X_2) $$ for $X_1,X_2,Y_1,Y_2$ with only invertible assumptions. Our proof shows the power of depth and the use of a non-linear activation function, since one layer network can only solve one equation,i.e.,$Y=WX$.