BLooP: Zero-Shot Abstractive Summarization using Large Language Models with Bigram Lookahead Promotion

TL;DR

BLooP is a training-free decoding method that enhances large language models' abstractive summarization by promoting bigram overlap with source documents, improving faithfulness and summary quality.

Contribution

We introduce BLooP, a novel decoding intervention that encourages bigram lookahead in LLMs without training or fine-tuning, significantly improving summarization faithfulness.

Findings

BLooP improves ROUGE and BARTScore metrics across multiple models and datasets.

Human evaluation confirms increased faithfulness without sacrificing readability.

Method requires no additional training or model modification.

Abstract

Abstractive summarization requires models to generate summaries that convey information in the source document. While large language models can generate summaries without fine-tuning, they often miss key details and include extraneous information. We propose BLooP (Bigram Lookahead Promotion), a simple training-free decoding intervention that encourages large language models (LLMs) to generate tokens that form bigrams from the source document. BLooP operates through a hash table lookup at each decoding step, requiring no training, fine-tuning, or model modification. We demonstrate improvements in ROUGE and BARTScore for Llama-3.1-8B-Instruct, Mistral-Nemo-Instruct-2407, and Gemma-2-9b-it on CNN/DM, CCSum, Multi-News, and SciTLDR. Human evaluation shows that BLooP significantly improves faithfulness without reducing readability. We make the code available at https://github.com/varuniyer/BLooP