A Qualitative Investigation into LLM-Generated Multilingual Code Comments and Automatic Evaluation Metrics

TL;DR

This study evaluates the multilingual capabilities of large language models in generating code comments, revealing significant challenges in accuracy and the unreliability of current evaluation metrics across diverse languages.

Contribution

It provides a comprehensive error taxonomy for multilingual code comments, assesses the reliability of evaluation metrics, and releases a large labeled dataset for future research.

Findings

Models often produce partially correct comments across languages.

Standard metrics fail to reliably distinguish correct from incorrect comments.

There is a significant overlap in metric scores between correct and incorrect comments.

Abstract

Large Language Models are essential coding assistants, yet their training is predominantly English-centric. In this study, we evaluate the performance of code language models in non-English contexts, identifying challenges in their adoption and integration into multilingual workflows. We conduct an open-coding study to analyze errors in code comments generated by five state-of-the-art code models, CodeGemma, CodeLlama, CodeQwen1.5, GraniteCode, and StarCoder2 across five natural languages: Chinese, Dutch, English, Greek, and Polish. Our study yields a dataset of 12,500 labeled generations, which we publicly release. We then assess the reliability of standard metrics in capturing comment \textit{correctness} across languages and evaluate their trustworthiness as judgment criteria. Through our open-coding investigation, we identified a taxonomy of 26 distinct error categories in model-generated code comments. They highlight variations in language cohesion, informativeness, and syntax adherence across different natural languages. Our analysis shows that, while these models frequently produce partially correct comments, modern neural metrics fail to reliably differentiate meaningful completions from random noise. Notably, the significant score overlap between expert-rated correct and incorrect comments calls into question the effectiveness of these metrics in assessing generated comments.