Can Model Fusing Help Transformers in Long Document Classification? An Empirical Study

TL;DR

This paper investigates whether model fusing techniques can improve transformer-based models' performance on long document classification tasks, addressing the sequence length limitations of models like BERT and Longformer.

Contribution

It provides an empirical comparison of model fusing approaches against standard transformer models for long document classification.

Findings

Model fusing shows potential to enhance long document classification accuracy.

Fused models outperform traditional transformers like BERT and Longformer in experiments.

The study offers insights into the benefits and limitations of model fusing in NLP tasks.

Abstract

Text classification is an area of research which has been studied over the years in Natural Language Processing (NLP). Adapting NLP to multiple domains has introduced many new challenges for text classification and one of them is long document classification. While state-of-the-art transformer models provide excellent results in text classification, most of them have limitations in the maximum sequence length of the input sequence. The majority of the transformer models are limited to 512 tokens, and therefore, they struggle with long document classification problems. In this research, we explore on employing Model Fusing for long document classification while comparing the results with well-known BERT and Longformer architectures.