A continuum among logarithmic, linear, and exponential functions, and its potential to improve generalization in neural networks

TL;DR

The paper introduces the soft exponential activation function, which smoothly interpolates among logarithmic, linear, and exponential functions, aiming to enhance neural network learning by enabling exact computation of various operations.

Contribution

It proposes a novel, differentiable activation function that can be trained within neural networks to perform a wide range of natural operations more precisely.

Findings

Potential to improve neural network generalization.

Enables exact computation of addition, multiplication, and other operations.

Flexible interpolation among key mathematical functions.

Abstract

We present the soft exponential activation function for artificial neural networks that continuously interpolates between logarithmic, linear, and exponential functions. This activation function is simple, differentiable, and parameterized so that it can be trained as the rest of the network is trained. We hypothesize that soft exponential has the potential to improve neural network learning, as it can exactly calculate many natural operations that typical neural networks can only approximate, including addition, multiplication, inner product, distance, polynomials, and sinusoids.