Kolmogorov-Arnold Networks in Low-Data Regimes: A Comparative Study with Multilayer Perceptrons

TL;DR

This study compares Kolmogorov-Arnold Networks and Multilayer Perceptrons in low-data scenarios, showing that MLPs with individualized activation functions outperform KANs in accuracy when data is scarce.

Contribution

Introduces a method for designing MLPs with unique activation functions per neuron, enabling a fairer comparison with KANs in low-data environments.

Findings

MLPs with personalized activation functions outperform KANs in accuracy with limited data.

In a three-class classification task, MLPs achieve 0.91 median accuracy versus 0.53 for KANs.

Model complexity trade-offs are crucial for effective learning in data-scarce settings.

Abstract

Multilayer Perceptrons (MLPs) have long been a cornerstone in deep learning, known for their capacity to model complex relationships. Recently, Kolmogorov-Arnold Networks (KANs) have emerged as a compelling alternative, utilizing highly flexible learnable activation functions directly on network edges, a departure from the neuron-centric approach of MLPs. However, KANs significantly increase the number of learnable parameters, raising concerns about their effectiveness in data-scarce environments. This paper presents a comprehensive comparative study of MLPs and KANs from both algorithmic and experimental perspectives, with a focus on low-data regimes. We introduce an effective technique for designing MLPs with unique, parameterized activation functions for each neuron, enabling a more balanced comparison with KANs. Using empirical evaluations on simulated data and two real-world data sets from medicine and engineering, we explore the trade-offs between model complexity and accuracy, with particular attention to the role of network depth. Our findings show that MLPs with individualized activation functions achieve significantly higher predictive accuracy with only a modest increase in parameters, especially when the sample size is limited to around one hundred. For example, in a three-class classification problem within additive manufacturing, MLPs achieve a median accuracy of 0.91, significantly outperforming KANs, which only reach a median accuracy of 0.53 with default hyperparameters. These results offer valuable insights into the impact of activation function selection in neural networks.