CombU: A Combined Unit Activation for Fitting Mathematical Expressions with Neural Networks

TL;DR

CombU introduces a novel approach by combining existing activation functions across layers, significantly improving neural networks' ability to fit complex mathematical expressions, as demonstrated on multiple datasets.

Contribution

The paper presents CombU, a new activation method that combines different functions across layers, enhancing the approximation of mathematical expressions beyond existing single-function approaches.

Findings

Outperforms all SOTA algorithms in 10 out of 16 metrics

Ranks in top three for remaining metrics

Proves theoretically effective for fitting mathematical expressions

Abstract

The activation functions are fundamental to neural networks as they introduce non-linearity into data relationships, thereby enabling deep networks to approximate complex data relations. Existing efforts to enhance neural network performance have predominantly focused on developing new mathematical functions. However, we find that a well-designed combination of existing activation functions within a neural network can also achieve this objective. In this paper, we introduce the Combined Units activation (CombU), which employs different activation functions at various dimensions across different layers. This approach can be theoretically proven to fit most mathematical expressions accurately. The experiments conducted on four mathematical expression datasets, compared against six State-Of-The-Art (SOTA) activation function algorithms, demonstrate that CombU outperforms all SOTA algorithms in 10 out of 16 metrics and ranks in the top three for the remaining six metrics.