Rethinking the Function of Neurons in KANs

TL;DR

This paper explores replacing the summation function in Kolmogorov-Arnold Networks with an average, leading to improved performance and training stability across various machine learning benchmarks.

Contribution

It introduces a simple modification to KAN neurons by using the average instead of the sum, which enhances practical utility and training stability.

Findings

Replacing sum with average improves performance.

The modification stabilizes training by controlling input range.

Empirical results show significant gains on benchmark tasks.

Abstract

The neurons of Kolmogorov-Arnold Networks (KANs) perform a simple summation motivated by the Kolmogorov-Arnold representation theorem, which asserts that sum is the only fundamental multivariate function. In this work, we investigate the potential for identifying an alternative multivariate function for KAN neurons that may offer increased practical utility. Our empirical research involves testing various multivariate functions in KAN neurons across a range of benchmark Machine Learning tasks. Our findings indicate that substituting the sum with the average function in KAN neurons results in significant performance enhancements compared to traditional KANs. Our study demonstrates that this minor modification contributes to the stability of training by confining the input to the spline within the effective range of the activation function. Our implementation and experiments are available at: \url{https://github.com/Ghaith81/dropkan}