Automatic Summarization of Long Documents

TL;DR

This paper presents three innovative algorithms enabling large language models to process extremely long documents exceeding 70,000 words, significantly improving summarization quality without changing the model architecture.

Contribution

The study introduces novel algorithms that overcome input size limitations of transformer models, allowing effective summarization of very long texts without architectural modifications.

Findings

Significant increase in BERTScore on long documents

Competitive ROUGE scores achieved

Algorithms enable processing of texts over 70,000 words

Abstract

A vast amount of textual data is added to the internet daily, making utilization and interpretation of such data difficult and cumbersome. As a result, automatic text summarization is crucial for extracting relevant information, saving precious reading time. Although many transformer-based models excel in summarization, they are constrained by their input size, preventing them from processing texts longer than their context size. This study introduces three novel algorithms that allow any LLM to efficiently overcome its input size limitation, effectively utilizing its full potential without any architectural modifications. We test our algorithms on texts with more than 70,000 words, and our experiments show a significant increase in BERTScore with competitive ROUGE scores.