Kolmogorov GAM Networks are all you need!

TL;DR

This paper introduces Kolmogorov GAM networks, an efficient additive architecture inspired by Kolmogorov's Superposition Theorem, offering a computationally attractive alternative to transformers for AI applications.

Contribution

The paper presents a novel K-GAM network architecture that combines topological embedding with additive models, reducing parameters and providing an alternative to transformer models.

Findings

K-GAM networks are computationally efficient and parallelizable.

They serve as a natural alternative to transformer architectures.

The approach is demonstrated on the Iris dataset.

Abstract

Kolmogorov GAM (K-GAM) networks are shown to be an efficient architecture for training and inference. They are an additive model with an embedding that is independent of the function of interest. They provide an alternative to the transformer architecture. They are the machine learning version of Kolmogorov's Superposition Theorem (KST) which provides an efficient representations of a multivariate function. Such representations have use in machine learning for encoding dictionaries (a.k.a. "look-up" tables). KST theory also provides a representation based on translates of the K\"oppen function. The goal of our paper is to interpret this representation in a machine learning context for applications in Artificial Intelligence (AI). Our architecture is equivalent to a topological embedding which is independent of the function together with an additive layer that uses a Generalized Additive Model (GAM). This provides a class of learning procedures with far fewer parameters than current deep learning algorithms. Implementation can be parallelizable which makes our algorithms computationally attractive. To illustrate our methodology, we use the Iris data from statistical learning. We also show that our additive model with non-linear embedding provides an alternative to transformer architectures which from a statistical viewpoint are kernel smoothers. Additive KAN models therefore provide a natural alternative to transformers. Finally, we conclude with directions for future research.