Improving Memory Efficiency for Training KANs via Meta Learning

TL;DR

This paper introduces MetaKANs, a meta-learning approach that generates weights for Kolmogorov-Arnold neural networks, significantly reducing memory usage and training costs while maintaining or improving performance across various tasks.

Contribution

The paper proposes MetaKANs, a novel meta-learning framework that enhances the memory efficiency and scalability of KANs without sacrificing interpretability or performance.

Findings

MetaKANs achieve comparable or better performance than standard KANs.

Significant reduction in trainable parameters and memory usage.

Effective across diverse tasks like symbolic regression, PDE solving, and image classification.

Abstract

Inspired by the Kolmogorov-Arnold representation theorem, KANs offer a novel framework for function approximation by replacing traditional neural network weights with learnable univariate functions. This design demonstrates significant potential as an efficient and interpretable alternative to traditional MLPs. However, KANs are characterized by a substantially larger number of trainable parameters, leading to challenges in memory efficiency and higher training costs compared to MLPs. To address this limitation, we propose to generate weights for KANs via a smaller meta-learner, called MetaKANs. By training KANs and MetaKANs in an end-to-end differentiable manner, MetaKANs achieve comparable or even superior performance while significantly reducing the number of trainable parameters and maintaining promising interpretability. Extensive experiments on diverse benchmark tasks, including symbolic regression, partial differential equation solving, and image classification, demonstrate the effectiveness of MetaKANs in improving parameter efficiency and memory usage. The proposed method provides an alternative technique for training KANs, that allows for greater scalability and extensibility, and narrows the training cost gap with MLPs stated in the original paper of KANs. Our code is available at https://github.com/Murphyzc/MetaKAN.