Analyzing the Performance of Large Language Models on Code Summarization

TL;DR

This paper investigates how large language models perform on code summarization tasks, highlighting the influence of token overlap between code and descriptions, and evaluating different metrics and code features.

Contribution

The study reveals the impact of token overlap, especially function names, on model performance and compares the effects of removing code structure versus function names.

Findings

Token overlap significantly affects model performance.

Removing function names reduces model performance.

BLEU and BERTScore are highly correlated metrics.

Abstract

Large language models (LLMs) such as Llama 2 perform very well on tasks that involve both natural language and source code, particularly code summarization and code generation. We show that for the task of code summarization, the performance of these models on individual examples often depends on the amount of (subword) token overlap between the code and the corresponding reference natural language descriptions in the dataset. This token overlap arises because the reference descriptions in standard datasets (corresponding to docstrings in large code bases) are often highly similar to the names of the functions they describe. We also show that this token overlap occurs largely in the function names of the code and compare the relative performance of these models after removing function names versus removing code structure. We also show that using multiple evaluation metrics like BLEU and BERTScore gives us very little additional insight since these metrics are highly correlated with each other.