The Fools are Certain; the Wise are Doubtful: Exploring LLM Confidence in Code Completion

TL;DR

This paper evaluates the confidence of large language models in code completion by analyzing perplexity across languages, models, and datasets, revealing language and model-dependent variations.

Contribution

It introduces a systematic analysis of LLM confidence in code generation using perplexity metrics across multiple languages and models, providing practical insights for developers.

Findings

Strongly-typed languages have lower perplexity than dynamically typed languages.

Scripting languages show higher perplexity, indicating lower model confidence.

Perplexity varies with the choice of LLM and language, affecting code completion reliability.

Abstract

Code completion entails the task of providing missing tokens given a surrounding context. It can boost developer productivity while providing a powerful code discovery tool. Following the Large Language Model (LLM) wave, code completion has been approached with diverse LLMs fine-tuned on code (code LLMs). The performance of code LLMs can be assessed with downstream and intrinsic metrics. Downstream metrics are usually employed to evaluate the practical utility of a model, but can be unreliable and require complex calculations and domain-specific knowledge. In contrast, intrinsic metrics such as perplexity, entropy, and mutual information, which measure model confidence or uncertainty, are simple, versatile, and universal across LLMs and tasks, and can serve as proxies for functional correctness and hallucination risk in LLM-generated code. Motivated by this, we evaluate the confidence of LLMs when generating code by measuring code perplexity across programming languages, models, and datasets using various LLMs, and a sample of 2254 files from 881 GitHub projects. We find that strongly-typed languages exhibit lower perplexity than dynamically typed languages. Scripting languages also demonstrate higher perplexity. Shell appears universally high in perplexity, whereas Java appears low. Code perplexity depends on the employed LLM; under a fixed model, relative language-level rankings are moderately stable across evaluation corpora. Although code comments often increase perplexity, the language ranking based on perplexity is barely affected by their presence. LLM researchers, developers, and users can employ our findings to assess the benefits and suitability of LLM-based code completion in specific software projects based on how language, model choice, and code characteristics impact model confidence.