GraphIC: A Graph-Based In-Context Example Retrieval Model for Multi-Step Reasoning

TL;DR

GraphIC introduces a graph-based retrieval method that improves in-context example selection for multi-step reasoning tasks, leading to better large language model performance by focusing on reasoning structures rather than superficial semantics.

Contribution

It proposes a novel reasoning-aware graph-based retrieval model that explicitly encodes reasoning steps and dependencies for more effective in-context example selection.

Findings

Outperforms 10 baseline methods across multiple reasoning tasks

Effectively filters superficial semantics to focus on reasoning processes

Enhances LLM performance in multi-step reasoning scenarios

Abstract

In-context learning (ICL) enhances large language models (LLMs) by incorporating demonstration examples, yet its effectiveness heavily depends on the quality of selected examples. Current methods typically use text embeddings to measure semantic similarity, which often introduces bias in multi-step reasoning tasks. This occurs because text embeddings contain irrelevant semantic information and lack deeper reasoning structures. To address this, we propose GraphIC, a graph-based retrieval model that leverages reasoning-aware representation and specialized similarity metric for in-context example retrieval. GraphIC first constructs thought graphs-directed, node-attributed graphs that explicitly model reasoning steps and their dependencies-for candidate examples and queries. This approach filters out superficial semantics while preserving essential reasoning processes. Next, GraphIC retrieves examples using a novel similarity metric tailored for these graphs, capturing sequential reasoning patterns and asymmetry between examples. Comprehensive evaluations across mathematical reasoning, code generation, and logical reasoning tasks demonstrate that GraphIC outperforms 10 baseline methods. Our results highlight the importance of reasoning-aware retrieval in ICL, offering a robust solution for enhancing LLM performance in multi-step reasoning scenarios.