Linear Classifier: An Often-Forgotten Baseline for Text Classification

TL;DR

This paper emphasizes the importance of using simple linear classifiers as baselines in text classification tasks to verify the effectiveness of advanced models like BERT, highlighting their competitive performance and robustness.

Contribution

It advocates for including linear classifiers as baselines in text classification to ensure the validity of results from advanced models like BERT.

Findings

Linear classifiers perform competitively on many text datasets.

Advanced models require proper application to achieve optimal results.

Simple baselines help validate the effectiveness of complex models.

Abstract

Large-scale pre-trained language models such as BERT are popular solutions for text classification. Due to the superior performance of these advanced methods, nowadays, people often directly train them for a few epochs and deploy the obtained model. In this opinion paper, we point out that this way may only sometimes get satisfactory results. We argue the importance of running a simple baseline like linear classifiers on bag-of-words features along with advanced methods. First, for many text data, linear methods show competitive performance, high efficiency, and robustness. Second, advanced models such as BERT may only achieve the best results if properly applied. Simple baselines help to confirm whether the results of advanced models are acceptable. Our experimental results fully support these points.