Scaling Up Summarization: Leveraging Large Language Models for Long Text Extractive Summarization

TL;DR

This paper presents EYEGLAXS, a framework that uses large language models like LLAMA2-7B and ChatGLM2-6B for efficient extractive summarization of long texts, achieving new performance benchmarks.

Contribution

Introduction of EYEGLAXS, a novel framework leveraging LLMs with PEFT and Flash Attention for extractive summarization, addressing computational challenges and setting new benchmarks.

Findings

EYEGLAXS outperforms existing methods on PubMed and ArXiv datasets.

Utilizes PEFT and Flash Attention for efficiency and scalability.

Demonstrates adaptability of LLMs to different sequence lengths.

Abstract

In an era where digital text is proliferating at an unprecedented rate, efficient summarization tools are becoming indispensable. While Large Language Models (LLMs) have been successfully applied in various NLP tasks, their role in extractive text summarization remains underexplored. This paper introduces EYEGLAXS (Easy Yet Efficient larGe LAnguage model for eXtractive Summarization), a framework that leverages LLMs, specifically LLAMA2-7B and ChatGLM2-6B, for extractive summarization of lengthy text documents. Instead of abstractive methods, which often suffer from issues like factual inaccuracies and hallucinations, EYEGLAXS focuses on extractive summarization to ensure factual and grammatical integrity. Utilizing state-of-the-art techniques such as Flash Attention and Parameter-Efficient Fine-Tuning (PEFT), EYEGLAXS addresses the computational and resource challenges typically associated with LLMs. The system sets new performance benchmarks on well-known datasets like PubMed and ArXiv. Furthermore, we extend our research through additional analyses that explore the adaptability of LLMs in handling different sequence lengths and their efficiency in training on smaller datasets. These contributions not only set a new standard in the field but also open up promising avenues for future research in extractive text summarization.