Measuring Determinism in Large Language Models for Software Code Review

TL;DR

This paper evaluates the consistency of large language models in generating software code review assessments, revealing variability even under controlled conditions, which raises concerns about their reliability in critical decision-making.

Contribution

The study systematically measures the test-retest reliability of four leading LLMs in code review tasks, highlighting inherent variability even with minimized randomness.

Findings

LLMs show variability in responses even at zero temperature

Response consistency varies across different models

Implications for using LLMs in critical software review processes

Abstract

Large Language Models (LLMs) promise to streamline software code reviews, but their ability to produce consistent assessments remains an open question. In this study, we tested four leading LLMs -- GPT-4o mini, GPT-4o, Claude 3.5 Sonnet, and LLaMA 3.2 90B Vision -- on 70 Java commits from both private and public repositories. By setting each model's temperature to zero, clearing context, and repeating the exact same prompts five times, we measured how consistently each model generated code-review assessments. Our results reveal that even with temperature minimized, LLM responses varied to different degrees. These findings highlight a consideration about the inherently limited consistency (test-retest reliability) of LLMs -- even when the temperature is set to zero -- and the need for caution when using LLM-generated code reviews to make real-world decisions.