Estimating Multiplicative Relations in Neural Networks

TL;DR

This paper introduces novel activation functions based on logarithmic properties to enable neural networks to learn multiplicative relations more effectively, addressing overfitting issues with product functions.

Contribution

The paper proposes a new pair of activation functions leveraging logarithms to translate products into linear forms, enhancing neural networks' ability to learn multiplicative functions.

Findings

Activation functions successfully model product relations.

Improved learning of multiplicative functions with reduced overfitting.

Potential generalization to complex arithmetic functions.

Abstract

Universal approximation theorem suggests that a shallow neural network can approximate any function. The input to neurons at each layer is a weighted sum of previous layer neurons and then an activation is applied. These activation functions perform very well when the output is a linear combination of input data. When trying to learn a function which involves product of input data, the neural networks tend to overfit the data to approximate the function. In this paper we will use properties of logarithmic functions to propose a pair of activation functions which can translate products into linear expression and learn using backpropagation. We will try to generalize this approach for some complex arithmetic functions and test the accuracy on a disjoint distribution with the training set.