A parametric activation function based on Wendland RBF

TL;DR

This paper proposes a new parametric activation function based on Wendland RBFs, offering improved stability, locality, and performance in deep neural networks, especially in regression tasks, by integrating classical approximation theory with modern deep learning.

Contribution

It introduces a novel Wendland RBF-based activation function with tunable properties, bridging classical approximation theory and deep learning to enhance neural network performance.

Findings

Achieves superior accuracy in regression tasks

Maintains computational efficiency during training

Demonstrates improved stability and generalization

Abstract

This paper introduces a novel parametric activation function based on Wendland radial basis functions (RBFs) for deep neural networks. Wendland RBFs, known for their compact support, smoothness, and positive definiteness in approximation theory, are adapted to address limitations of traditional activation functions like ReLU, sigmoid, and tanh. The proposed enhanced Wendland activation combines a standard Wendland component with linear and exponential terms, offering tunable locality, improved gradient propagation, and enhanced stability during training. Theoretical analysis highlights its mathematical properties, including smoothness and adaptability, while empirical experiments on synthetic tasks (e.g., sine wave approximation) and benchmark datasets (MNIST, Fashion-MNIST) demonstrate competitive performance. Results show that the Wendland-based activation achieves superior accuracy in certain scenarios, particularly in regression tasks, while maintaining computational efficiency. The study bridges classical RBF theory with modern deep learning, suggesting that Wendland activations can mitigate overfitting and improve generalization through localized, smooth transformations. Future directions include hybrid architectures and domain-specific adaptations.