Mathematical analysis of one-layer neural network with fixed biases, a new activation function and other observations

TL;DR

This paper provides a rigorous mathematical analysis of a one-layer neural network with fixed biases and a new activation function, demonstrating convergence and spectral bias properties.

Contribution

It introduces a detailed theoretical framework for analyzing convergence and spectral bias in simple neural networks, including a new activation function called FReX.

Findings

Proves convergence of gradient descent with squared loss in the model.

Establishes the spectral bias property for the learning process.

Proposes and discusses the convergence of an alternative activation function, FReX.

Abstract

We analyze a simple one-hidden-layer neural network with ReLU activation functions and fixed biases, with one-dimensional input and output. We study both continuous and discrete versions of the model, and we rigorously prove the convergence of the learning process with the $L^2$ squared loss function and the gradient descent procedure. We also prove the spectral bias property for this learning process. Several conclusions of this analysis are discussed; in particular, regarding the structure and properties that activation functions should possess, as well as the relationships between the spectrum of certain operators and the learning process. Based on this, we also propose an alternative activation function, the full-wave rectified exponential function (FReX), and we discuss the convergence of the gradient descent with this alternative activation function.