A Tutorial on Neural Networks and Gradient-free Training

TL;DR

This paper provides a matrix-based, tutorial-style overview of neural networks, explaining their mathematical structure and comparing gradient-based and gradient-free training methods.

Contribution

It introduces a compact matrix representation of neural networks and analyzes both gradient-based and gradient-free training approaches.

Findings

Gradient-free methods are compared with gradient-based training.

Neural networks are represented as compositions of linear and nonlinear functions.

The paper discusses convergence and accuracy of different training methods.

Abstract

This paper presents a compact, matrix-based representation of neural networks in a self-contained tutorial fashion. Specifically, we develop neural networks as a composition of several vector-valued functions. Although neural networks are well-understood pictorially in terms of interconnected neurons, neural networks are mathematical nonlinear functions constructed by composing several vector-valued functions. Using basic results from linear algebra, we represent a neural network as an alternating sequence of linear maps and scalar nonlinear functions, also known as activation functions. The training of neural networks requires the minimization of a cost function, which in turn requires the computation of a gradient. Using basic multivariable calculus results, the cost gradient is also shown to be a function composed of a sequence of linear maps and nonlinear functions. In addition to the analytical gradient computation, we consider two gradient-free training methods and compare the three training methods in terms of convergence rate and prediction accuracy.