Do BERT-Like Bidirectional Models Still Perform Better on Text Classification in the Era of LLMs?

TL;DR

This paper compares BERT-like models and LLMs in text classification, finding BERT-like models often outperform LLMs on certain tasks, and proposes a task-specific model selection strategy.

Contribution

It systematically evaluates BERT-like models versus LLMs across multiple datasets and introduces TaMAS, a task-driven model selection approach.

Findings

BERT-like models outperform LLMs on pattern-driven tasks.

LLMs excel in tasks requiring deep semantics or world knowledge.

TaMAS improves task-specific model selection.

Abstract

The rapid adoption of LLMs has overshadowed the potential advantages of traditional BERT-like models in text classification. This study challenges the prevailing "LLM-centric" trend by systematically comparing three category methods, i.e., BERT-like models fine-tuning, LLM internal state utilization, and zero-shot inference across six high-difficulty datasets. Our findings reveal that BERT-like models often outperform LLMs. We further categorize datasets into three types, perform PCA and probing experiments, and identify task-specific model strengths: BERT-like models excel in pattern-driven tasks, while LLMs dominate those requiring deep semantics or world knowledge. Based on this, we propose TaMAS, a fine-grained task selection strategy, advocating for a nuanced, task-driven approach over a one-size-fits-all reliance on LLMs.