On the Convergence of (Stochastic) Gradient Descent for Kolmogorov--Arnold Networks

TL;DR

This paper provides the first theoretical convergence guarantees for gradient descent and stochastic gradient descent when training Kolmogorov--Arnold Networks, explaining their empirical success in various scientific and machine learning tasks.

Contribution

It establishes global convergence results for GD and SGD on two-layer KANs, including physics-informed variants, using neural tangent kernel analysis.

Findings

GD achieves global linear convergence for large hidden dimensions

SGD converges in expectation for regression tasks

Convergence guarantees extend to physics-informed KANs

Abstract

Kolmogorov--Arnold Networks (KANs), a recently proposed neural network architecture, have gained significant attention in the deep learning community, due to their potential as a viable alternative to multi-layer perceptrons (MLPs) and their broad applicability to various scientific tasks. Empirical investigations demonstrate that KANs optimized via stochastic gradient descent (SGD) are capable of achieving near-zero training loss in various machine learning (e.g., regression, classification, and time series forecasting, etc.) and scientific tasks (e.g., solving partial differential equations). In this paper, we provide a theoretical explanation for the empirical success by conducting a rigorous convergence analysis of gradient descent (GD) and SGD for two-layer KANs in solving both regression and physics-informed tasks. For regression problems, we establish using the neural tangent kernel perspective that GD achieves global linear convergence of the objective function when the hidden dimension of KANs is sufficiently large. We further extend these results to SGD, demonstrating a similar global convergence in expectation. Additionally, we analyze the global convergence of GD and SGD for physics-informed KANs, which unveils additional challenges due to the more complex loss structure. This is the first work establishing the global convergence guarantees for GD and SGD applied to optimize KANs and physics-informed KANs.