Residual Kolmogorov-Arnold Network for Enhanced Deep Learning

TL;DR

This paper introduces RKAN, a compact plug-in module that enhances deep CNNs by integrating polynomial feature transformations, leading to improved performance and efficiency across various vision benchmarks.

Contribution

The paper proposes a novel Residual Kolmogorov-Arnold Network (RKAN) module that can be integrated into existing CNNs to improve their learning capacity and performance.

Findings

RKAN improves baseline CNN performance on multiple vision benchmarks.

RKAN enhances learning efficiency and reduces overfitting risks.

RKAN achieves state-of-the-art results in several vision tasks.

Abstract

Despite their immense success, deep convolutional neural networks (CNNs) can be difficult to optimize and costly to train due to hundreds of layers within the network depth. Conventional convolutional operations are fundamentally limited by their linear nature along with fixed activations, where many layers are needed to learn meaningful patterns in data. Because of the sheer size of these networks, this approach is simply computationally inefficient, and poses overfitting or gradient explosion risks, especially in small datasets. As a result, we introduce a "plug-in" module, called Residual Kolmogorov-Arnold Network (RKAN). Our module is highly compact, so it can be easily added into any stage (level) of traditional deep networks, where it learns to integrate supportive polynomial feature transformations to existing convolutional frameworks. RKAN offers consistent improvements over baseline models in different vision tasks and widely tested benchmarks, accomplishing cutting-edge performance on them.