Variational Neural Networks: Every Layer and Neuron Can Be Unique

TL;DR

This paper introduces variational neural networks where each layer and neuron can have a unique activation function, optimized through gradient descent to improve neural network performance.

Contribution

It proposes a novel framework representing activation functions as linear combinations of candidates, with derived gradient formulas for optimization.

Findings

Activation functions can be optimized per neuron.

Gradient formulas enable efficient training of variational neural networks.

Potential for improved neural network performance.

Abstract

The choice of activation function can significantly influence the performance of neural networks. The lack of guiding principles for the selection of activation function is lamentable. We try to address this issue by introducing our variational neural networks, where the activation function is represented as a linear combination of possible candidate functions, and an optimal activation is obtained via minimization of a loss function using gradient descent method. The gradient formulae for the loss function with respect to these expansion coefficients are central for the implementation of gradient descent algorithm, and here we derive these gradient formulae.