Comparative Study of Long Document Classification

TL;DR

This study systematically compares various machine learning and transformer-based models for long document classification across multiple datasets, highlighting that simpler models often perform competitively with BERT, especially when computational resources are limited.

Contribution

It provides an exhaustive benchmarking of traditional and modern models on standard long document datasets, emphasizing the effectiveness of simple algorithms.

Findings

Basic algorithms perform competitively with BERT on most datasets.

BERT models perform consistently well across datasets.

Simple models like BiLSTM + Max are effective for long document classification.

Abstract

The amount of information stored in the form of documents on the internet has been increasing rapidly. Thus it has become a necessity to organize and maintain these documents in an optimum manner. Text classification algorithms study the complex relationships between words in a text and try to interpret the semantics of the document. These algorithms have evolved significantly in the past few years. There has been a lot of progress from simple machine learning algorithms to transformer-based architectures. However, existing literature has analyzed different approaches on different data sets thus making it difficult to compare the performance of machine learning algorithms. In this work, we revisit long document classification using standard machine learning approaches. We benchmark approaches ranging from simple Naive Bayes to complex BERT on six standard text classification datasets. We present an exhaustive comparison of different algorithms on a range of long document datasets. We re-iterate that long document classification is a simpler task and even basic algorithms perform competitively with BERT-based approaches on most of the datasets. The BERT-based models perform consistently well on all the datasets and can be blindly used for the document classification task when the computations cost is not a concern. In the shallow model's category, we suggest the usage of raw BiLSTM + Max architecture which performs decently across all the datasets. Even simpler Glove + Attention bag of words model can be utilized for simpler use cases. The importance of using sophisticated models is clearly visible in the IMDB sentiment dataset which is a comparatively harder task.