From Misuse to Mastery: Enhancing Code Generation with Knowledge-Driven AI Chaining

TL;DR

This paper introduces KPC, a knowledge-driven prompt chaining method that significantly improves the quality and reliability of code generated by LLMs, especially in exception handling, through iterative check-rewrite steps.

Contribution

It proposes a novel knowledge-driven prompt chaining approach for enhancing code generation, addressing exception handling challenges in LLMs with empirical validation.

Findings

Achieves 109.86% and 578.57% improvements in static evaluation metrics.

Reduces 18 runtime bugs in sampled code.

Demonstrates effectiveness in managing exceptions in generated code.

Abstract

Large Language Models (LLMs) have shown promising results in automatic code generation by improving coding efficiency to a certain extent. However, generating high-quality and reliable code remains a formidable task because of LLMs' lack of good programming practice, especially in exception handling. In this paper, we first conduct an empirical study and summarise three crucial challenges of LLMs in exception handling, i.e., incomplete exception handling, incorrect exception handling and abuse of try-catch. We then try prompts with different granularities to address such challenges, finding fine-grained knowledge-driven prompts works best. Based on our empirical study, we propose a novel Knowledge-driven Prompt Chaining-based code generation approach, name KPC, which decomposes code generation into an AI chain with iterative check-rewrite steps and chains fine-grained knowledge-driven prompts to assist LLMs in considering exception-handling specifications. We evaluate our KPC-based approach with 3,079 code generation tasks extracted from the Java official API documentation. Extensive experimental results demonstrate that the KPC-based approach has considerable potential to ameliorate the quality of code generated by LLMs. It achieves this through proficiently managing exceptions and obtaining remarkable enhancements of 109.86% and 578.57% with static evaluation methods, as well as a reduction of 18 runtime bugs in the sampled dataset with dynamic validation.