Sinusoidal Approximation Theorem for Kolmogorov-Arnold Networks

TL;DR

This paper introduces a novel Kolmogorov-Arnold Network variant using learnable sinusoidal activations, providing theoretical proof of validity and demonstrating competitive performance with traditional neural networks in approximating multivariable functions.

Contribution

It proposes a new sinusoidal approximation approach within KANs, replacing inner and outer functions with learnable frequencies and fixed phases, supported by a theoretical proof.

Findings

Outperforms fixed-frequency Fourier transform methods

Achieves performance comparable to multilayer perceptrons

Provides theoretical validation for the sinusoidal KAN variant

Abstract

The Kolmogorov-Arnold representation theorem states that any continuous multivariable function can be exactly represented as a finite superposition of continuous single variable functions. Subsequent simplifications of this representation involve expressing these functions as parameterized sums of a smaller number of unique monotonic functions. These developments led to the proof of the universal approximation capabilities of multilayer perceptron networks with sigmoidal activations, forming the alternative theoretical direction of most modern neural networks. Kolmogorov-Arnold Networks (KANs) have been recently proposed as an alternative to multilayer perceptrons. KANs feature learnable nonlinear activations applied directly to input values, modeled as weighted sums of basis spline functions. This approach replaces the linear transformations and sigmoidal post-activations used in traditional perceptrons. Subsequent works have explored alternatives to spline-based activations. In this work, we propose a novel KAN variant by replacing both the inner and outer functions in the Kolmogorov-Arnold representation with weighted sinusoidal functions of learnable frequencies. Inspired by simplifications introduced by Lorentz and Sprecher, we fix the phases of the sinusoidal activations to linearly spaced constant values and provide a proof of its theoretical validity. We also conduct numerical experiments to evaluate its performance on a range of multivariable functions, comparing it with fixed-frequency Fourier transform methods and multilayer perceptrons (MLPs). We show that it outperforms the fixed-frequency Fourier transform and achieves comparable performance to MLPs.