KKANs: Kurkova-Kolmogorov-Arnold Networks and Their Learning Dynamics

TL;DR

This paper introduces KKANs, a novel neural network architecture inspired by mathematical theorems, demonstrating superior approximation capabilities and detailed analysis of their learning dynamics and geometric complexity.

Contribution

The paper proposes KKAN, a new universal approximator architecture combining MLPs and basis functions, with comprehensive theoretical and empirical validation across multiple scientific machine learning tasks.

Findings

KKANs outperform MLPs and KANs in function approximation and operator learning.

KKANs achieve performance comparable to optimized MLPs in physics-informed learning.

Analysis reveals universal learning stages and the correlation between geometric complexity and SNR.

Abstract

Inspired by the Kolmogorov-Arnold representation theorem and Kurkova's principle of using approximate representations, we propose the Kurkova-Kolmogorov-Arnold Network (KKAN), a new two-block architecture that combines robust multi-layer perceptron (MLP) based inner functions with flexible linear combinations of basis functions as outer functions. We first prove that KKAN is a universal approximator, and then we demonstrate its versatility across scientific machine-learning applications, including function regression, physics-informed machine learning (PIML), and operator-learning frameworks. The benchmark results show that KKANs outperform MLPs and the original Kolmogorov-Arnold Networks (KANs) in function approximation and operator learning tasks and achieve performance comparable to fully optimized MLPs for PIML. To better understand the behavior of the new representation models, we analyze their geometric complexity and learning dynamics using information bottleneck theory, identifying three universal learning stages, fitting, transition, and diffusion, across all types of architectures. We find a strong correlation between geometric complexity and signal-to-noise ratio (SNR), with optimal generalization achieved during the diffusion stage. Additionally, we propose self-scaled residual-based attention weights to maintain high SNR dynamically, ensuring uniform convergence and prolonged learning.