Theoretical Exploration of Flexible Transmitter Model

TL;DR

This paper provides a theoretical analysis of the Flexible Transmitter (FT) neural network model, demonstrating its universal approximation capabilities, potential for reduced complexity, and favorable optimization landscape.

Contribution

It offers the first theoretical exploration of FTNet, showing its universal approximation, exponential complexity reduction, and absence of local minima.

Findings

FTNet is a universal approximator.

Approximation complexity of FTNet can be exponentially smaller.

Any local minimum of FTNet is a global minimum.

Abstract

Neural network models generally involve two important components, i.e., network architecture and neuron model. Although there are abundant studies about network architectures, only a few neuron models have been developed, such as the MP neuron model developed in 1943 and the spiking neuron model developed in the 1950s. Recently, a new bio-plausible neuron model, Flexible Transmitter (FT) model, has been proposed. It exhibits promising behaviors, particularly on temporal-spatial signals, even when simply embedded into the common feedforward network architecture. This paper attempts to understand the properties of the FT network (FTNet) theoretically. Under mild assumptions, we show that: i) FTNet is a universal approximator; ii) the approximation complexity of FTNet can be exponentially smaller than those of commonly-used real-valued neural networks with feedforward/recurrent architectures and is of the same order in the worst case; iii) any local minimum of FTNet is the global minimum, implying that it is possible to identify global minima by local search algorithms.