Key-Element-Informed sLLM Tuning for Document Summarization

TL;DR

This paper introduces KEITSum, a method that improves small-scale LLMs' document summarization by focusing on key elements, resulting in more relevant summaries with fewer hallucinations, comparable to larger proprietary models.

Contribution

The paper presents KEITSum, a novel instruction tuning approach that enhances sLLMs' ability to generate relevant summaries by emphasizing key document elements.

Findings

KEITSum improves relevance and reduces hallucinations in sLLM summaries.

sLLMs with KEITSum achieve performance comparable to proprietary LLMs.

The method is effective on dialogue and news datasets.

Abstract

Remarkable advances in large language models (LLMs) have enabled high-quality text summarization. However, this capability is currently accessible only through LLMs of substantial size or proprietary LLMs with usage fees. In response, smaller-scale LLMs (sLLMs) of easy accessibility and low costs have been extensively studied, yet they often suffer from missing key information and entities, i.e., low relevance, in particular, when input documents are long. We hence propose a key-element-informed instruction tuning for summarization, so-called KEITSum, which identifies key elements in documents and instructs sLLM to generate summaries capturing these key elements. Experimental results on dialogue and news datasets demonstrate that sLLM with KEITSum indeed provides high-quality summarization with higher relevance and less hallucinations, competitive to proprietary LLM.