PRKAN: Parameter-Reduced Kolmogorov-Arnold Networks

TL;DR

PRKANs introduce parameter reduction techniques to Kolmogorov-Arnold Networks, achieving comparable or superior performance to traditional models like MLPs on standard datasets, with benefits from GRBFs and layer normalization.

Contribution

This work presents PRKANs, a novel approach to significantly reduce parameters in KAN layers, enhancing their practicality and performance across neural network architectures.

Findings

PRKANs outperform existing KAN variants on MNIST and Fashion-MNIST.

Attention-augmented PRKANs match MLP performance with longer training.

Incorporating GRBFs and layer normalization improves KAN design.

Abstract

Kolmogorov-Arnold Networks (KANs) represent an innovation in neural network architectures, offering a compelling alternative to Multi-Layer Perceptrons (MLPs) in models such as Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Transformers. By advancing network design, KANs drive groundbreaking research and enable transformative applications across various scientific domains involving neural networks. However, existing KANs often require significantly more parameters in their network layers than MLPs. To address this limitation, this paper introduces PRKANs (Parameter-Reduced Kolmogorov-Arnold Networks), which employ several methods to reduce the parameter count in KAN layers, making them comparable to MLP layers. Experimental results on the MNIST and Fashion-MNIST datasets demonstrate that PRKANs outperform several existing KANs, and their variant with attention mechanisms rivals the performance of MLPs, albeit with slightly longer training times. Furthermore, the study highlights the advantages of Gaussian Radial Basis Functions (GRBFs) and layer normalization in KAN designs. The repository for this work is available at: https://github.com/hoangthangta/All-KAN.