MetaCluster: Enabling Deep Compression of Kolmogorov-Arnold Network

TL;DR

MetaCluster is a novel framework that significantly compresses Kolmogorov-Arnold Networks by clustering basis coefficient vectors, reducing storage by up to 80 times without accuracy loss.

Contribution

It introduces a meta-learner-guided clustering approach to make KANs highly compressible while maintaining their expressivity and accuracy.

Findings

Achieves up to 80x parameter reduction on image datasets.

Maintains accuracy despite aggressive compression.

Effective on high-dimensional equation modeling tasks.

Abstract

Kolmogorov-Arnold Networks (KANs) replace scalar weights with per-edge vectors of basis coefficients, thereby increasing expressivity and accuracy while also resulting in a multiplicative increase in parameters and memory. We propose MetaCluster, a framework that makes KANs highly compressible without sacrificing accuracy. Specifically, a lightweight meta-learner, trained jointly with the KAN, maps low-dimensional embeddings to coefficient vectors, thereby shaping them to lie on a low-dimensional manifold that is amenable to clustering. We then run K-means in coefficient space and replace per-edge vectors with shared centroids. Afterwards, the meta-learner can be discarded, and a brief fine-tuning of the centroid codebook recovers any residual accuracy loss. The resulting model stores only a small codebook and per-edge indices, exploiting the vector nature of KAN parameters to amortize storage across multiple coefficients. On MNIST, CIFAR-10, and CIFAR-100, across standard KANs and ConvKANs using multiple basis functions, MetaCluster achieves a reduction of up to $80\times$ in parameter storage, with no loss in accuracy. Similarly, on high-dimensional equation modeling tasks, MetaCluster achieves a parameter reduction of $124.1\times$, without impacting performance. Code will be released upon publication.