Automatic Grid Updates for Kolmogorov-Arnold Networks using Layer Histograms

TL;DR

This paper introduces an automatic domain update method for Kolmogorov-Arnold Networks using layer histograms, improving training efficiency and enabling out-of-distribution detection, while maintaining or surpassing prior performance.

Contribution

The paper presents a novel histogram-based algorithm that allows KAN layers to autonomously update their domain grids during training, reducing user overhead and enhancing OOD detection.

Findings

AdaptKAN matches or exceeds prior KAN and MLP performance on four tasks.

The histogram algorithm effectively detects out-of-distribution inputs.

Automatic domain updates improve training efficiency and model robustness.

Abstract

Kolmogorov-Arnold Networks (KANs) are a class of neural networks that have received increased attention in recent literature. In contrast to MLPs, KANs leverage parameterized, trainable activation functions and offer several benefits including improved interpretability and higher accuracy on learning symbolic equations. However, the original KAN architecture requires adjustments to the domain discretization of the network (called the "domain grid") during training, creating extra overhead for the user in the training process. Typical KAN layers are not designed with the ability to autonomously update their domains in a data-driven manner informed by the changing output ranges of previous layers. As an added benefit, this histogram algorithm may also be applied towards detecting out-of-distribution (OOD) inputs in a variety of settings. We demonstrate that AdaptKAN exceeds or matches the performance of prior KAN architectures and MLPs on four different tasks: learning scientific equations from the Feynman dataset, image classification from frozen features, learning a control Lyapunov function, and detecting OOD inputs on the OpenOOD v1.5 benchmark.