Beyond Weighted Summation: Learnable Nonlinear Aggregation Functions for Robust Artificial Neurons

TL;DR

This paper explores learnable nonlinear aggregation functions in artificial neurons to enhance robustness against noisy inputs, demonstrating improved performance on CIFAR-10 with hybrid neurons that adaptively combine linear and nonlinear aggregation.

Contribution

Introduces differentiable nonlinear aggregation mechanisms and hybrid neurons that interpolate between linear and nonlinear aggregation for improved robustness.

Findings

Hybrid neurons significantly improve noise robustness.

Learned aggregation parameters favor sub-linear and high novelty utilization.

Modest gains observed on clean data with F-Mean hybrids.

Abstract

Weighted summation has remained the default input aggregation mechanism in artificial neurons since the earliest neural network models. While computationally efficient, this design implicitly behaves like a mean-based estimator and is therefore sensitive to noisy or extreme inputs. This paper investigates whether replacing fixed linear aggregation with learnable nonlinear alternatives can improve neural network robustness without sacrificing trainability. Two differentiable aggregation mechanisms are introduced: an F-Mean neuron based on a learnable power-weighted aggregation rule, and a Gaussian Support neuron based on distance-aware affinity weighting. To preserve the optimisation stability of standard neurons, hybrid neurons are proposed that interpolate between linear and nonlinear aggregation through a learnable blending parameter. Evaluated in multilayer perceptrons and convolutional neural networks on CIFAR-10 and a noisy CIFAR-10 variant with additive Gaussian corruption, hybrid neurons consistently improve robustness under noise while F-Mean hybrids also yield modest gains on clean data. The three-way hybrid achieves robustness scores of up to 0.991 compared to 0.890 for the standard baseline, and learned parameters converge consistently to sub-linear aggregation (p $\approx$ 0.43--0.50) and high novelty utilisation ($\alpha$ $\approx$ 0.69--0.79). These findings suggest that neuron-level aggregation is a meaningful and underexplored design dimension for building more noise-tolerant neural networks.