Graph of Records: Boosting Retrieval Augmented Generation for Long-context Summarization with Graphs

TL;DR

This paper introduces Graph of Records (GoR), a novel method that enhances retrieval-augmented generation for long-context summarization by leveraging historical LLM responses through graph neural networks, leading to improved summarization performance.

Contribution

The paper proposes GoR, which constructs a graph from retrieved texts and LLM responses, using GNNs and BERTScore-based training to better utilize historical responses for long-context summarization.

Findings

GoR outperforms 12 baselines on four datasets.

Achieves up to 19% improvement in Rouge scores.

Effectively leverages historical responses for better summaries.

Abstract

Retrieval-augmented generation (RAG) has revitalized Large Language Models (LLMs) by injecting non-parametric factual knowledge. Compared with long-context LLMs, RAG is considered an effective summarization tool in a more concise and lightweight manner, which can interact with LLMs multiple times using diverse queries to get comprehensive responses. However, the LLM-generated historical responses, which contain potentially insightful information, are largely neglected and discarded by existing approaches, leading to suboptimal results. In this paper, we propose $\textit{graph of records}$ ($\textbf{GoR}$), which leverages historical responses generated by LLMs to enhance RAG for long-context global summarization. Inspired by the $\textit{retrieve-then-generate}$ paradigm of RAG, we construct a graph by establishing an edge between the retrieved text chunks and the corresponding LLM-generated response. To further uncover the intricate correlations between them, GoR features a $\textit{graph neural network}$ and an elaborately designed $\textit{BERTScore}$-based objective for self-supervised model training, enabling seamless supervision signal backpropagation between reference summaries and node embeddings. We comprehensively compare GoR with 12 baselines across four long-context summarization datasets, and the results indicate that our proposed method reaches the best performance ($\textit{e.g.}$, 15%, 8%, and 19% improvement over retrievers w.r.t. Rouge-L, Rouge-1, and Rouge-2 on the WCEP dataset). Extensive experiments further demonstrate the effectiveness of GoR.