The Behavior of Large Language Models When Prompted to Generate Code Explanations

TL;DR

This study examines how large language models generate code explanations for beginner programming examples, revealing factors influencing explanation quality and consistency across languages.

Contribution

It systematically analyzes the effects of prompts, code, language, and model parameters on LLM-generated code explanations, highlighting patterns and limitations.

Findings

Explanations vary significantly based on prompt wording and model parameters.

Java and Python explanations have consistent readability and lexical density.

Explanations are generally correct but often lack completeness, conciseness, and specificity.

Abstract

This paper systematically investigates the generation of code explanations by Large Language Models (LLMs) for code examples commonly encountered in introductory programming courses. Our findings reveal significant variations in the nature of code explanations produced by LLMs, influenced by factors such as the wording of the prompt, the specific code examples under consideration, the programming language involved, the temperature parameter, and the version of the LLM. However, a consistent pattern emerges for Java and Python, where explanations exhibit a Flesch-Kincaid readability level of approximately 7-8 grade and a consistent lexical density, indicating the proportion of meaningful words relative to the total explanation size. Additionally, the generated explanations consistently achieve high scores for correctness, but lower scores on three other metrics: completeness, conciseness, and specificity.