GrowOVER: How Can LLMs Adapt to Growing Real-World Knowledge?

TL;DR

This paper introduces GrowOVER, a dynamic benchmarking framework for open-domain QA and dialogue that adapts to evolving knowledge, and proposes a retrieval-interactive language model approach to improve knowledge updating without retraining.

Contribution

The paper presents GrowOVER benchmarks for continuous knowledge updates and a novel retrieval-interactive framework that enhances LLMs' ability to adapt to new information without retraining.

Findings

Retrieval-augmented models struggle with untrained or outdated knowledge.

The proposed framework improves performance over existing methods.

Our approach matches or exceeds the performance of continuously trained models.

Abstract

In the real world, knowledge is constantly evolving, which can render existing knowledge-based datasets outdated. This unreliability highlights the critical need for continuous updates to ensure both accuracy and relevance in knowledge-intensive tasks. To address this, we propose GrowOVER-QA and GrowOVER-Dialogue, dynamic open-domain QA and dialogue benchmarks that undergo a continuous cycle of updates, keeping pace with the rapid evolution of knowledge. Our research indicates that retrieval-augmented language models (RaLMs) struggle with knowledge that has not been trained on or recently updated. Consequently, we introduce a novel retrieval-interactive language model framework, where the language model evaluates and reflects on its answers for further re-retrieval. Our exhaustive experiments demonstrate that our training-free framework significantly improves upon existing methods, performing comparably to or even surpassing continuously trained language models.