A preliminary study on continual learning in computer vision using Kolmogorov-Arnold Networks

TL;DR

This study evaluates Kolmogorov-Arnold Networks (KAN) for continual learning in computer vision, demonstrating their potential to outperform traditional MLPs and original KAN models in class-incremental tasks on MNIST datasets.

Contribution

The paper extends KAN evaluation to computer vision, introduces an efficient KAN variant, and analyzes hyperparameters and convolutional KAN performance in continual learning.

Findings

Efficient KAN outperforms traditional MLPs and original KAN.

Hyperparameters significantly influence model performance.

Preliminary convolutional KAN results compare favorably with CNNs.

Abstract

Deep learning has long been dominated by multi-layer perceptrons (MLPs), which have demonstrated superiority over other optimizable models in various domains. Recently, a new alternative to MLPs has emerged - Kolmogorov-Arnold Networks (KAN)- which are based on a fundamentally different mathematical framework. According to their authors, KANs address several major issues in MLPs, such as catastrophic forgetting in continual learning scenarios. However, this claim has only been supported by results from a regression task on a toy 1D dataset. In this paper, we extend the investigation by evaluating the performance of KANs in continual learning tasks within computer vision, specifically using the MNIST datasets. To this end, we conduct a structured analysis of the behavior of MLPs and two KAN-based models in a class-incremental learning scenario, ensuring that the architectures involved have the same number of trainable parameters. Our results demonstrate that an efficient version of KAN outperforms both traditional MLPs and the original KAN implementation. We further analyze the influence of hyperparameters in MLPs and KANs, as well as the impact of certain trainable parameters in KANs, such as bias and scale weights. Additionally, we provide a preliminary investigation of recent KAN-based convolutional networks and compare their performance with that of traditional convolutional neural networks. Our codes can be found at https://github.com/MrPio/KAN-Continual_Learning_tests.