A Comparison of SVM against Pre-trained Language Models (PLMs) for Text Classification Tasks

TL;DR

This study compares the effectiveness of pre-trained language models versus traditional SVM classifiers with TFIDF features for text classification, finding that SVM can often outperform PLMs in both cost and accuracy.

Contribution

The paper provides a systematic comparison showing that simple SVM classifiers can match or outperform complex PLMs in text classification tasks, especially in domain-specific contexts.

Findings

PLMs do not significantly outperform SVMs on tested datasets.

SVM with TFIDF features can be more cost-effective and sometimes more accurate.

Traditional methods remain competitive against modern PLMs in certain NLP tasks.

Abstract

The emergence of pre-trained language models (PLMs) has shown great success in many Natural Language Processing (NLP) tasks including text classification. Due to the minimal to no feature engineering required when using these models, PLMs are becoming the de facto choice for any NLP task. However, for domain-specific corpora (e.g., financial, legal, and industrial), fine-tuning a pre-trained model for a specific task has shown to provide a performance improvement. In this paper, we compare the performance of four different PLMs on three public domain-free datasets and a real-world dataset containing domain-specific words, against a simple SVM linear classifier with TFIDF vectorized text. The experimental results on the four datasets show that using PLMs, even fine-tuned, do not provide significant gain over the linear SVM classifier. Hence, we recommend that for text classification tasks, traditional SVM along with careful feature engineering can pro-vide a cheaper and superior performance than PLMs.