MLP-KAN: Unifying Deep Representation and Function Learning

TL;DR

MLP-KAN is a unified deep learning framework that combines representation and function learning within a Mixture-of-Experts architecture, eliminating manual model selection and adapting dynamically to diverse tasks.

Contribution

This paper introduces MLP-KAN, a novel integrated approach that unifies representation and function learning using MoE architecture within a transformer framework.

Findings

Achieves superior versatility across multiple datasets

Delivers competitive performance in both learning paradigms

Simplifies model selection process for diverse tasks

Abstract

Recent advancements in both representation learning and function learning have demonstrated substantial promise across diverse domains of artificial intelligence. However, the effective integration of these paradigms poses a significant challenge, particularly in cases where users must manually decide whether to apply a representation learning or function learning model based on dataset characteristics. To address this issue, we introduce MLP-KAN, a unified method designed to eliminate the need for manual model selection. By integrating Multi-Layer Perceptrons (MLPs) for representation learning and Kolmogorov-Arnold Networks (KANs) for function learning within a Mixture-of-Experts (MoE) architecture, MLP-KAN dynamically adapts to the specific characteristics of the task at hand, ensuring optimal performance. Embedded within a transformer-based framework, our work achieves remarkable results on four widely-used datasets across diverse domains. Extensive experimental evaluation demonstrates its superior versatility, delivering competitive performance across both deep representation and function learning tasks. These findings highlight the potential of MLP-KAN to simplify the model selection process, offering a comprehensive, adaptable solution across various domains. Our code and weights are available at \url{https://github.com/DLYuanGod/MLP-KAN}.