RAS: Retrieval-And-Structuring for Knowledge-Intensive LLM Generation

TL;DR

The paper introduces RAS, a framework that dynamically constructs question-specific knowledge graphs through iterative retrieval and structuring, significantly improving reasoning in large language models on knowledge-intensive tasks.

Contribution

RAS is a novel method that combines retrieval planning with incremental graph construction to enhance reasoning in LLMs by organizing retrieved information into structured knowledge graphs.

Findings

RAS outperforms baselines on seven benchmarks.

Achieves up to 8.7% and 7.0% improvements with different LLMs.

Dynamic structuring improves reasoning accuracy and robustness.

Abstract

Large language models (LLMs) have achieved impressive performance on knowledge-intensive tasks, yet they often struggle with multi-step reasoning due to the unstructured nature of retrieved context. While retrieval-augmented generation (RAG) methods provide external information, the lack of explicit organization among retrieved passages limits their effectiveness, leading to brittle reasoning pathways. Recent interpretability studies highlighting the importance of structured intermediate reasoning further align with this perspective. We propose Retrieval-And-Structuring (RAS), a framework that dynamically constructs question-specific knowledge graphs through iterative retrieval and structured knowledge building. RAS interleaves targeted retrieval planning with incremental graph construction, enabling models to assemble and reason over evolving knowledge structures tailored to each query. On seven knowledge-intensive benchmarks, RAS consistently outperforms strong baselines, achieving up to 8.7\% and 7.0\% gains with proprietary and open-source LLMs, respectively. Our results demonstrate that dynamic, question-specific knowledge structuring offers a robust path to improving reasoning accuracy and robustness in language model generation.