FourierKAN outperforms MLP on Text Classification Head Fine-tuning

TL;DR

This paper demonstrates that Fourier KAN, a novel classification head, surpasses traditional MLPs in accuracy and efficiency for text classification tasks when used with pre-trained transformers.

Contribution

Introduction of Fourier KAN as an effective, lightweight alternative to MLP heads for fine-tuning pre-trained language models in text classification.

Findings

FR-KAN outperforms MLP by 10% in accuracy and 11% in F1-score.

FR-KAN is more computationally efficient and trains faster.

Results consistent across multiple models and datasets.

Abstract

In resource constraint settings, adaptation to downstream classification tasks involves fine-tuning the final layer of a classifier (i.e. classification head) while keeping rest of the model weights frozen. Multi-Layer Perceptron (MLP) heads fine-tuned with pre-trained transformer backbones have long been the de facto standard for text classification head fine-tuning. However, the fixed non-linearity of MLPs often struggles to fully capture the nuances of contextual embeddings produced by pre-trained models, while also being computationally expensive. In our work, we investigate the efficacy of KAN and its variant, Fourier KAN (FR-KAN), as alternative text classification heads. Our experiments reveal that FR-KAN significantly outperforms MLPs with an average improvement of 10% in accuracy and 11% in F1-score across seven pre-trained transformer models and four text classification tasks. Beyond performance gains, FR-KAN is more computationally efficient and trains faster with fewer parameters. These results underscore the potential of FR-KAN to serve as a lightweight classification head, with broader implications for advancing other Natural Language Processing (NLP) tasks.