KG-Augmented Executable CoT for Mathematical Coding

TL;DR

KGA-ECoT enhances mathematical reasoning and code generation in large language models by integrating knowledge graphs, structured problem decomposition, and executable code, leading to significant accuracy improvements on benchmarks.

Contribution

This paper introduces KGA-ECoT, a novel framework combining knowledge graphs and executable code to improve complex reasoning in language models, a significant advancement over prior prompting methods.

Findings

Achieves 10+ percentage point accuracy improvements on benchmarks.

Demonstrates the effectiveness of knowledge retrieval via GraphRAG.

Validates the importance of external code execution for accuracy.

Abstract

In recent years, large language models (LLMs) have excelled in natural language processing tasks but face significant challenges in complex reasoning tasks such as mathematical reasoning and code generation. To address these limitations, we propose KG-Augmented Executable Chain-of-Thought (KGA-ECoT), a novel framework that enhances code generation through knowledge graphs and improves mathematical reasoning via executable code. KGA-ECoT decomposes problems into a Structured Task Graph, leverages efficient GraphRAG for precise knowledge retrieval from mathematical libraries, and generates verifiable code to ensure computational accuracy. Evaluations on multiple mathematical reasoning benchmarks demonstrate that KGA-ECoT significantly outperforms existing prompting methods, achieving absolute accuracy improvements ranging from several to over ten percentage points. Further analysis confirms the critical roles of GraphRAG in enhancing code quality and external code execution in ensuring precision. These findings collectively establish KGA-ECoT as a robust and highly generalizable framework for complex mathematical reasoning tasks.