A Comparison of LSTM and BERT for Small Corpus

TL;DR

This paper compares LSTM and BERT models on small datasets for intent classification, finding that simpler LSTM models outperform BERT in accuracy and training time, emphasizing task and data considerations in model selection.

Contribution

It provides empirical evidence that traditional LSTM models can outperform BERT on small datasets, challenging the assumption that larger pre-trained models are always superior in such scenarios.

Findings

LSTM outperforms BERT on small intent classification datasets.

LSTM models train faster than BERT in small data settings.

Model choice should consider task and data characteristics.

Abstract

Recent advancements in the NLP field showed that transfer learning helps with achieving state-of-the-art results for new tasks by tuning pre-trained models instead of starting from scratch. Transformers have made a significant improvement in creating new state-of-the-art results for many NLP tasks including but not limited to text classification, text generation, and sequence labeling. Most of these success stories were based on large datasets. In this paper we focus on a real-life scenario that scientists in academia and industry face frequently: given a small dataset, can we use a large pre-trained model like BERT and get better results than simple models? To answer this question, we use a small dataset for intent classification collected for building chatbots and compare the performance of a simple bidirectional LSTM model with a pre-trained BERT model. Our experimental results show that bidirectional LSTM models can achieve significantly higher results than a BERT model for a small dataset and these simple models get trained in much less time than tuning the pre-trained counterparts. We conclude that the performance of a model is dependent on the task and the data, and therefore before making a model choice, these factors should be taken into consideration instead of directly choosing the most popular model.