Improving Multi-step RAG with Hypergraph-based Memory for Long-Context Complex Relational Modeling

TL;DR

This paper introduces HGMem, a hypergraph-based memory system that enhances multi-step retrieval-augmented generation by modeling high-order correlations for better reasoning and global understanding.

Contribution

The paper presents a novel hypergraph-based memory mechanism that captures complex relations and improves multi-step reasoning in RAG systems.

Findings

HGMem significantly outperforms baseline systems on global sense-making tasks.

The hypergraph memory enhances reasoning depth and coherence.

Experiments demonstrate improved performance across diverse datasets.

Abstract

Multi-step retrieval-augmented generation (RAG) has become a widely adopted strategy for enhancing large language models (LLMs) on tasks that demand global comprehension and intensive reasoning. Many RAG systems incorporate a working memory module to consolidate retrieved information. However, existing memory designs function primarily as passive storage that accumulates isolated facts for the purpose of condensing the lengthy inputs and generating new sub-queries through deduction. This static nature overlooks the crucial high-order correlations among primitive facts, the compositions of which can often provide stronger guidance for subsequent steps. Therefore, their representational strength and impact on multi-step reasoning and knowledge evolution are limited, resulting in fragmented reasoning and weak global sense-making capacity in extended contexts. We introduce HGMem, a hypergraph-based memory mechanism that extends the concept of memory beyond simple storage into a dynamic, expressive structure for complex reasoning and global understanding. In our approach, memory is represented as a hypergraph whose hyperedges correspond to distinct memory units, enabling the progressive formation of higher-order interactions within memory. This mechanism connects facts and thoughts around the focal problem, evolving into an integrated and situated knowledge structure that provides strong propositions for deeper reasoning in subsequent steps. We evaluate HGMem on several challenging datasets designed for global sense-making. Extensive experiments and in-depth analyses show that our method consistently improves multi-step RAG and substantially outperforms strong baseline systems across diverse tasks.