Pragmatic Reasoning improves LLM Code Generation

TL;DR

This paper introduces CodeRSA, a pragmatic reasoning framework based on the Rational Speech Act model, which improves language-to-code generation in large language models by handling ambiguities and multiple instruction interpretations, leading to better performance.

Contribution

It extends pragmatic reasoning to natural language-to-code tasks and proposes a method to manage multiple equivalent instructions, enhancing code generation quality in LLMs.

Findings

CodeRSA outperforms baseline models on HumanEval and MBPP benchmarks.

It surpasses state-of-the-art approaches in most cases.

Qualitative analysis confirms the model's reasoning aligns with human-like understanding.

Abstract

Pragmatic reasoning is pervasive in human-human communication - it allows us to leverage shared knowledge and counterfactual reasoning in order to infer the intention of a conversational partner given their ambiguous or underspecified message. In human-computer communication, underspecified messages often represent a major challenge: for instance, translating natural language instructions into code is difficult when user instructions contain inherent ambiguities. In the present paper, we aim to scale up the pragmatic "Rational Speech Act" framework to naturalistic language-to-code problems, and propose a way of dealing with multiple meaning-equivalent instruction alternatives, an issue that does not arise in previous toy-scale problems. We evaluate our method, CodeRSA, with two recent LLMs (Llama-3-8B-Instruct and Qwen-2.5-7B-Instruct) on two widely used code generation benchmarks (HumanEval and MBPP). Our experimental results show that CodeRSA consistently outperforms common baselines, surpasses the state-of-the-art approach in most cases, and demonstrates robust overall performance. Qualitative analyses demonstrate that it exhibits the desired behavior for the right reasons. These findings underscore the effectiveness of integrating pragmatic reasoning into a naturalistic complex communication task, language-to-code generation, offering a promising direction for enhancing code generation quality in LLMs and emphasizing the importance of pragmatic reasoning in complex communication settings.