Concurrent training methods for Kolmogorov-Arnold networks: Disjoint datasets and FPGA implementation

TL;DR

This paper proposes concurrency-driven training enhancements for Kolmogorov-Arnold networks, achieving over 40x faster training on CPUs and FPGA implementation, by introducing pre-training, disjoint data training, and FPGA parallelisation.

Contribution

It introduces novel concurrency techniques for KAN training, including data disjoint training, pre-training, and FPGA parallelisation, significantly accelerating the process.

Findings

KANs trained with new methods are over 40 times faster than neural networks.

Parallelisation on FPGAs is successfully implemented and tested.

Experimental results are fully reproducible with available source code.

Abstract

The present paper introduces concurrency-driven enhancements to the training algorithm for the Kolmogorov-Arnold networks (KANs) that is based on the Newton-Kaczmarz (NK) method. Prior research shows that KANs trained using the NK-based approach significantly overtake classical neural networks based on multilayer perceptrons (MLPs) in terms of accuracy and training time. Although some parts of the algorithm, such as the evaluation of the basis functions, can be parallelised, the fundamental limitation lies in the sequential computation of the updates - each update depends on the results of the previous step, obstructing parallelisation. However, substantial acceleration is achievable. Three complementary strategies are proposed in the present paper: (i) a pre-training procedure tailored to the NK updates' structure, (ii) training on disjoint subsets of data, followed by models' merging, not in the context of federated learning, but as a mechanism for accelerating the convergence, and (iii) a parallelisation technique suitable for execution on field-programmable gate arrays (FPGAs), which is implemented and tested directly on the device. With these novel techniques, computational experiments show that KANs can be trained more than 40 times faster than neural networks, when training is done to the same accuracy on CPUs. All presented experimental results are fully reproducible, with the complete source codes available online.