Guiding Language Models of Code with Global Context using Monitors

TL;DR

This paper introduces monitor-guided decoding (MGD), a method that uses static analysis to improve the global contextual understanding of language models of code, leading to better accuracy and fewer hallucinations across multiple languages.

Contribution

The paper presents MGD, a novel approach that integrates static analysis as monitors during decoding to enhance code generation by language models, especially for unseen global context.

Findings

MGD improves compilation rates and ground truth agreement.

Smaller LMs with MGD outperform larger models without it.

MGD generalizes across multiple programming languages and constraints.

Abstract

Language models of code (LMs) work well when the surrounding code provides sufficient context. This is not true when it becomes necessary to use types, functionality or APIs defined elsewhere in the repository or a linked library, especially those not seen during training. LMs suffer from limited awareness of such global context and end up hallucinating. Integrated development environments (IDEs) assist developers in understanding repository context using static analysis. We extend this assistance, enjoyed by developers, to LMs. We propose monitor-guided decoding (MGD) where a monitor uses static analysis to guide the decoding. We construct a repository-level dataset PragmaticCode for method-completion in Java and evaluate MGD on it. On models of varying parameter scale, by monitoring for type-consistent object dereferences, MGD consistently improves compilation rates and agreement with ground truth. Further, LMs with fewer parameters, when augmented with MGD, can outperform larger LMs. With MGD, SantaCoder-1.1B achieves better compilation rate and next-identifier match than the much larger text-davinci-003 model. We also conduct a generalizability study to evaluate the ability of MGD to generalize to multiple programming languages (Java, C# and Rust), coding scenarios (e.g., correct number of arguments to method calls), and to enforce richer semantic constraints (e.g., stateful API protocols). Our data and implementation are available at https://github.com/microsoft/monitors4codegen .