FABLE: Forest-Based Adaptive Bi-Path LLM-Enhanced Retrieval for Multi-Document Reasoning

TL;DR

FABLE introduces a forest-based, hierarchical retrieval framework that enhances multi-document reasoning in LLMs by combining structured knowledge organization with adaptive, efficient evidence gathering, outperforming existing retrieval methods.

Contribution

FABLE presents a novel hierarchical forest index and bi-path retrieval strategy integrating LLM guidance, improving multi-document reasoning efficiency and accuracy over traditional flat retrieval methods.

Findings

Outperforms state-of-the-art RAG methods in accuracy.

Achieves up to 94% token reduction compared to full-context inference.

Maintains comparable accuracy to full-context LLMs with structured retrieval.

Abstract

The rapid expansion of long-context Large Language Models (LLMs) has reignited debate on whether Retrieval-Augmented Generation (RAG) remains necessary. However, empirical evidence reveals persistent limitations of long-context inference, including the lost-in-the-middle phenomenon, high computational cost, and poor scalability for multi-document reasoning. Conversely, traditional RAG systems, while efficient, are constrained by flat chunk-level retrieval that introduces semantic noise and fails to support structured cross-document synthesis. We present \textbf{FABLE}, a \textbf{F}orest-based \textbf{A}daptive \textbf{B}i-path \textbf{L}LM-\textbf{E}nhanced retrieval framework that integrates LLMs into both knowledge organization and retrieval. FABLE constructs LLM-enhanced hierarchical forest indexes with multi-granularity semantic structures, then employs a bi-path strategy combining LLM-guided hierarchical traversal with structure-aware propagation for fine-grained evidence acquisition, with explicit budget control for adaptive efficiency trade-offs. Extensive experiments demonstrate that FABLE consistently outperforms SOTA RAG methods and achieves comparable accuracy to full-context LLM inference with up to 94\% token reduction, showing that long-context LLMs amplify rather than fully replace the need for structured retrieval.