Retrieval Meets Reasoning: Dynamic In-Context Editing for Long-Text Understanding

TL;DR

This paper presents a novel dynamic in-context editing approach that enhances large language models' ability to perform multi-hop reasoning over long texts by interactively gathering and integrating relevant information, surpassing existing methods.

Contribution

The paper introduces a dynamic in-context editing technique that treats lengthy contexts as editable external knowledge, improving reasoning in LLMs without extensive retraining.

Findings

Outperforms state-of-the-art context window extrapolation methods

Enables Llama2 to perform multi-hop reasoning effectively

Reduces training and computational costs for long-text reasoning

Abstract

Current Large Language Models (LLMs) face inherent limitations due to their pre-defined context lengths, which impede their capacity for multi-hop reasoning within extensive textual contexts. While existing techniques like Retrieval-Augmented Generation (RAG) have attempted to bridge this gap by sourcing external information, they fall short when direct answers are not readily available. We introduce a novel approach that re-imagines information retrieval through dynamic in-context editing, inspired by recent breakthroughs in knowledge editing. By treating lengthy contexts as malleable external knowledge, our method interactively gathers and integrates relevant information, thereby enabling LLMs to perform sophisticated reasoning steps. Experimental results demonstrate that our method effectively empowers context-limited LLMs, such as Llama2, to engage in multi-hop reasoning with improved performance, which outperforms state-of-the-art context window extrapolation methods and even compares favorably to more advanced commercial long-context models. Our interactive method not only enhances reasoning capabilities but also mitigates the associated training and computational costs, making it a pragmatic solution for enhancing LLMs' reasoning within expansive contexts.