Free-Knots Kolmogorov-Arnold Network: On the Analysis of Spline Knots and Advancing Stability

TL;DR

This paper introduces a novel Free Knots Kolmogorov-Arnold Network that improves training stability, reduces parameters, and enhances smoothness of learned splines, demonstrating its effectiveness across diverse datasets and tasks.

Contribution

It proposes a new Free Knots KAN with fewer parameters, a training strategy for $C^2$ continuity, and comprehensive evaluation showing improved stability and performance.

Findings

Enhanced training stability and smoother activation functions.

Reduced number of trainable parameters to match MLP scale.

Effective across multiple data domains and tasks.

Abstract

Kolmogorov-Arnold Neural Networks (KANs) have gained significant attention in the machine learning community. However, their implementation often suffers from poor training stability and heavy trainable parameter. Furthermore, there is limited understanding of the behavior of the learned activation functions derived from B-splines. In this work, we analyze the behavior of KANs through the lens of spline knots and derive the lower and upper bound for the number of knots in B-spline-based KANs. To address existing limitations, we propose a novel Free Knots KAN that enhances the performance of the original KAN while reducing the number of trainable parameters to match the trainable parameter scale of standard Multi-Layer Perceptrons (MLPs). Additionally, we introduce new a training strategy to ensure $C^2$ continuity of the learnable spline, resulting in smoother activation compared to the original KAN and improve the training stability by range expansion. The proposed method is comprehensively evaluated on 8 datasets spanning various domains, including image, text, time series, multimodal, and function approximation tasks. The promising results demonstrates the feasibility of KAN-based network and the effectiveness of proposed method.