Context-Augmented Code Generation Using Programming Knowledge Graphs

TL;DR

This paper introduces Programming Knowledge Graphs (PKG) to improve code generation by enhancing retrieval accuracy and reducing hallucinations in Large Language Models, leading to significant performance gains.

Contribution

The paper presents a novel PKG framework with tree pruning and re-ranking mechanisms that significantly improve code generation accuracy over existing retrieval-augmented methods.

Findings

Up to 20% increase in pass@1 accuracy on HumanEval.

34% improvement over baselines on MBPP.

Effective reduction of hallucinations in code generation.

Abstract

Large Language Models (LLMs) excel at code generation but struggle with complex problems. Retrieval-Augmented Generation (RAG) mitigates this issue by integrating external knowledge, yet retrieval models often miss relevant context, and generation models hallucinate with irrelevant data. We propose Programming Knowledge Graph (PKG) for semantic representation and fine-grained retrieval of code and text. Our approach enhances retrieval precision through tree pruning and mitigates hallucinations via a re-ranking mechanism that integrates non-RAG solutions. Structuring external data into finer-grained nodes improves retrieval granularity. Evaluations on HumanEval and MBPP show up to 20% pass@1 accuracy gains and a 34% improvement over baselines on MBPP. Our findings demonstrate that our proposed PKG approach along with re-ranker effectively address complex problems while maintaining minimal negative impact on solutions that are already correct without RAG. The replication package is published at https://github.com/iamshahd/ProgrammingKnowledgeGraph