SineKAN: Kolmogorov-Arnold Networks Using Sinusoidal Activation Functions

TL;DR

SineKAN introduces sinusoidal activation functions into Kolmogorov-Arnold Networks, achieving comparable or better performance than spline-based KANs with increased speed and potential for scalable accuracy in vision tasks.

Contribution

The paper proposes replacing B-Spline activation functions with re-weighted sine functions in KANs, demonstrating improved speed and competitive performance.

Findings

SineKAN performs better or comparable to B-Spline KAN models.

SineKAN achieves substantial speed increases over baseline KANs.

SineKAN's numerical accuracy scales similarly to dense neural networks.

Abstract

Recent work has established an alternative to traditional multi-layer perceptron neural networks in the form of Kolmogorov-Arnold Networks (KAN). The general KAN framework uses learnable activation functions on the edges of the computational graph followed by summation on nodes. The learnable edge activation functions in the original implementation are basis spline functions (B-Spline). Here, we present a model in which learnable grids of B-Spline activation functions are replaced by grids of re-weighted sine functions (SineKAN). We evaluate numerical performance of our model on a benchmark vision task. We show that our model can perform better than or comparable to B-Spline KAN models and an alternative KAN implementation based on periodic cosine and sine functions representing a Fourier Series. Further, we show that SineKAN has numerical accuracy that could scale comparably to dense neural networks (DNNs). Compared to the two baseline KAN models, SineKAN achieves a substantial speed increase at all hidden layer sizes, batch sizes, and depths. Current advantage of DNNs due to hardware and software optimizations are discussed along with theoretical scaling. Additionally, properties of SineKAN compared to other KAN implementations and current limitations are also discussed