Studying How Configurations Impact Code Generation in LLMs: the Case of ChatGPT

TL;DR

This paper systematically investigates how ChatGPT's configuration parameters, especially top-p and temperature, affect Java code generation performance, providing practical recommendations for optimal settings.

Contribution

It offers a detailed analysis of parameter impacts on code generation quality and introduces guidelines to mitigate non-determinism in LLM-based coding tools.

Findings

Top-p significantly influences code quality, more than temperature.

Increasing prompt repetitions improves output consistency.

Creativity levels can enhance or hinder code accuracy.

Abstract

Leveraging LLMs for code generation is becoming increasingly common, as tools like ChatGPT can suggest method implementations with minimal input, such as a method signature and brief description. Empirical studies further highlight the effectiveness of LLMs in handling such tasks, demonstrating notable performance in code generation scenarios. However, LLMs are inherently non-deterministic, with their output influenced by parameters such as temperature, which regulates the model's level of creativity, and top-p, which controls the choice of the tokens that shall appear in the output. Despite their significance, the role of these parameters is often overlooked. This paper systematically studies the impact of these parameters, as well as the number of prompt repetitions required to account for non-determinism, in the context of 548 Java methods. We observe significantly different performances across different configurations of ChatGPT, with temperature having a marginal impact compared to the more prominent influence of the top-p parameter. Additionally, we show how creativity can enhance code generation tasks. Finally, we provide concrete recommendations for addressing the non-determinism of the model.