Logit Reweighting for Topic-Focused Summarization

TL;DR

This paper introduces a lightweight logit reweighting technique to improve topic focus in abstractive summarization, offering a resource-efficient alternative to fine-tuning that enhances topical relevance without sacrificing summary quality.

Contribution

The paper proposes a novel logit reweighting method for topic-focused summarization, demonstrating its effectiveness across different models and maintaining summary quality.

Findings

Reweighting logits increases topic-relevant vocabulary usage.

Threshold Selection improves topical focus without quality loss.

Methods are resource-efficient compared to fine-tuning.

Abstract

Generating abstractive summaries that adhere to a specific topic remains a significant challenge for language models. While standard approaches, such as fine-tuning, are resource-intensive, simpler methods like prompt engineering often struggle to maintain topical focus, particularly with smaller models. To address this, we propose a lightweight method that enhances topical relevance by directly reweighting the logits of topic-relevant tokens during generation. We evaluate three such reweighting techniques: Constant Shift, which adds a constant value to logits; Factor Scaling, which multiplies them by a factor; and Threshold Selection, which selectively boosts logits that exceed a probability threshold. Experiments on the NEWTS topical summarization dataset, using both Gemma-2B and Llama-3-8B models, show that these techniques effectively increase the use of topic-relevant vocabulary. Notably, the Threshold Selection method successfully improves topical focus without compromising summary quality-a trade-off often seen in other approaches. Our findings demonstrate that directly reweighting logits is a practical and resource-efficient alternative to fine-tuning, offering a promising pathway for precisely controlling the thematic content of generated text.