Beyond Relevant Documents: A Knowledge-Intensive Approach for Query-Focused Summarization using Large Language Models

TL;DR

This paper introduces a knowledge-intensive query-focused summarization method that combines retrieval and LLM-based summarization, enabling effective summaries without pre-existing relevant documents, demonstrated through a new dataset and extensive experiments.

Contribution

It presents a novel framework integrating retrieval and large language models for query-focused summarization, addressing limitations of traditional methods that depend on pre-existing relevant documents.

Findings

Outperforms traditional methods in generating accurate, relevant summaries.

Effectively retrieves relevant documents from large-scale knowledge bases.

Demonstrates versatility across diverse query scenarios.

Abstract

Query-focused summarization (QFS) is a fundamental task in natural language processing with broad applications, including search engines and report generation. However, traditional approaches assume the availability of relevant documents, which may not always hold in practical scenarios, especially in highly specialized topics. To address this limitation, we propose a novel knowledge-intensive approach that reframes QFS as a knowledge-intensive task setup. This approach comprises two main components: a retrieval module and a summarization controller. The retrieval module efficiently retrieves potentially relevant documents from a large-scale knowledge corpus based on the given textual query, eliminating the dependence on pre-existing document sets. The summarization controller seamlessly integrates a powerful large language model (LLM)-based summarizer with a carefully tailored prompt, ensuring the generated summary is comprehensive and relevant to the query. To assess the effectiveness of our approach, we create a new dataset, along with human-annotated relevance labels, to facilitate comprehensive evaluation covering both retrieval and summarization performance. Extensive experiments demonstrate the superior performance of our approach, particularly its ability to generate accurate summaries without relying on the availability of relevant documents initially. This underscores our method's versatility and practical applicability across diverse query scenarios.