Recursive Abstractive Processing for Retrieval in Dynamic Datasets

TL;DR

This paper introduces a new algorithm for maintaining recursive-abstractive retrieval structures in dynamic datasets and a post-retrieval method to enhance context quality, improving retrieval performance in real-world scenarios.

Contribution

It presents a novel, efficient algorithm for updating hierarchical retrieval structures in dynamic datasets and a query-focused recursive abstractive post-processing technique.

Findings

Effective handling of dynamic datasets with hierarchical retrieval structures

Significant improvement in retrieval quality through recursive abstractive post-processing

Validated on real-world datasets demonstrating enhanced performance

Abstract

Recent retrieval-augmented models enhance basic methods by building a hierarchical structure over retrieved text chunks through recursive embedding, clustering, and summarization. The most relevant information is then retrieved from both the original text and generated summaries. However, such approaches face limitations with dynamic datasets, where adding or removing documents over time complicates the updating of hierarchical representations formed through clustering. We propose a new algorithm to efficiently maintain the recursive-abstractive tree structure in dynamic datasets, without compromising performance. Additionally, we introduce a novel post-retrieval method that applies query-focused recursive abstractive processing to substantially improve context quality. Our method overcomes the limitations of other approaches by functioning as a black-box post-retrieval layer compatible with any retrieval algorithm. Both algorithms are validated through extensive experiments on real-world datasets, demonstrating their effectiveness in handling dynamic data and improving retrieval performance.