Reduced Effectiveness of Kolmogorov-Arnold Networks on Functions with Noise

TL;DR

This paper investigates how noise impacts the performance of Kolmogorov-Arnold Networks (KAN) and proposes oversampling combined with denoising via kernel filtering to mitigate noise effects, revealing limitations in data volume and filtering optimization.

Contribution

It introduces a combined oversampling and kernel filtering approach to reduce noise impact on KAN, and quantitatively analyzes the performance degradation with noisy data.

Findings

Performance degrades with added noise, following a specific trend as data volume increases.

Oversampling and filtering can mitigate noise effects but have cost and optimization challenges.

Noise ultimately limits the effectiveness of KAN despite mitigation strategies.

Abstract

It has been observed that even a small amount of noise introduced into the dataset can significantly degrade the performance of KAN. In this brief note, we aim to quantitatively evaluate the performance when noise is added to the dataset. We propose an oversampling technique combined with denoising to alleviate the impact of noise. Specifically, we employ kernel filtering based on diffusion maps for pre-filtering the noisy data for training KAN network. Our experiments show that while adding i.i.d. noise with any fixed SNR, when we increase the amount of training data by a factor of $r$, the test-loss (RMSE) of KANs will exhibit a performance trend like $\text{test-loss} \sim \mathcal{O}(r^{-\frac{1}{2}})$ as $r\to +\infty$. We conclude that applying both oversampling and filtering strategies can reduce the detrimental effects of noise. Nevertheless, determining the optimal variance for the kernel filtering process is challenging, and enhancing the volume of training data substantially increases the associated costs, because the training dataset needs to be expanded multiple times in comparison to the initial clean data. As a result, the noise present in the data ultimately diminishes the effectiveness of Kolmogorov-Arnold networks.