On Training of Kolmogorov-Arnold Networks

TL;DR

This paper investigates the training dynamics of Kolmogorov-Arnold Networks (KANs), comparing them with MLPs, and offers insights and recommendations to improve their training stability on high-dimensional datasets.

Contribution

It provides a comprehensive analysis of KAN training, compares them with MLPs, and suggests methods to enhance training stability for larger models.

Findings

KANs are effective alternatives to MLPs on high-dimensional data

KANs have better parameter efficiency than MLPs

Training of KANs is more unstable but can be improved

Abstract

Kolmogorov-Arnold Networks have recently been introduced as a flexible alternative to multi-layer Perceptron architectures. In this paper, we examine the training dynamics of different KAN architectures and compare them with corresponding MLP formulations. We train with a variety of different initialization schemes, optimizers, and learning rates, as well as utilize back propagation free approaches like the HSIC Bottleneck. We find that (when judged by test accuracy) KANs are an effective alternative to MLP architectures on high-dimensional datasets and have somewhat better parameter efficiency, but suffer from more unstable training dynamics. Finally, we provide recommendations for improving training stability of larger KAN models.