Bridging KAN and MLP: MJKAN, a Hybrid Architecture with Both Efficiency and Expressiveness

TL;DR

This paper introduces MJKAN, a hybrid neural network layer combining KAN and MLP features, which enhances expressiveness and efficiency, validated across diverse benchmarks with promising results.

Contribution

MJKAN is a novel hybrid layer integrating FiLM-like modulation with RBF activations, addressing KAN limitations and improving practical neural network performance.

Findings

MJKAN outperforms MLPs in function regression tasks.

Performance depends on the number of basis functions used.

Smaller basis sizes improve generalization in classification tasks.

Abstract

Kolmogorov-Arnold Networks (KANs) have garnered attention for replacing fixed activation functions with learnable univariate functions, but they exhibit practical limitations, including high computational costs and performance deficits in general classification tasks. In this paper, we propose the Modulation Joint KAN (MJKAN), a novel neural network layer designed to overcome these challenges. MJKAN integrates a FiLM (Feature-wise Linear Modulation)-like mechanism with Radial Basis Function (RBF) activations, creating a hybrid architecture that combines the non-linear expressive power of KANs with the efficiency of Multilayer Perceptrons (MLPs). We empirically validated MJKAN's performance across a diverse set of benchmarks, including function regression, image classification (MNIST, CIFAR-10/100), and natural language processing (AG News, SMS Spam). The results demonstrate that MJKAN achieves superior approximation capabilities in function regression tasks, significantly outperforming MLPs, with performance improving as the number of basis functions increases. Conversely, in image and text classification, its performance was competitive with MLPs but revealed a critical dependency on the number of basis functions. We found that a smaller basis size was crucial for better generalization, highlighting that the model's capacity must be carefully tuned to the complexity of the data to prevent overfitting. In conclusion, MJKAN offers a flexible architecture that inherits the theoretical advantages of KANs while improving computational efficiency and practical viability.