Reimagining Linear Probing: Kolmogorov-Arnold Networks in Transfer Learning

TL;DR

This paper proposes Kolmogorov-Arnold Networks (KAN) as a flexible, spline-based alternative to linear probing in transfer learning, demonstrating improved accuracy and generalization on image classification tasks.

Contribution

Introducing KAN as a novel, spline-based method to enhance linear probing in transfer learning, allowing modeling of complex data relationships.

Findings

KAN outperforms traditional linear probing in accuracy.

KAN improves generalization across configurations.

Hyperparameter tuning enhances KAN's performance.

Abstract

This paper introduces Kolmogorov-Arnold Networks (KAN) as an enhancement to the traditional linear probing method in transfer learning. Linear probing, often applied to the final layer of pre-trained models, is limited by its inability to model complex relationships in data. To address this, we propose substituting the linear probing layer with KAN, which leverages spline-based representations to approximate intricate functions. In this study, we integrate KAN with a ResNet-50 model pre-trained on ImageNet and evaluate its performance on the CIFAR-10 dataset. We perform a systematic hyperparameter search, focusing on grid size and spline degree (k), to optimize KAN's flexibility and accuracy. Our results demonstrate that KAN consistently outperforms traditional linear probing, achieving significant improvements in accuracy and generalization across a range of configurations. These findings indicate that KAN offers a more powerful and adaptable alternative to conventional linear probing techniques in transfer learning.