Automatic Code Summarization via ChatGPT: How Far Are We?

TL;DR

This paper evaluates ChatGPT's effectiveness in automatic Python code summarization, comparing it with state-of-the-art models using standard metrics, and discusses its limitations and future challenges.

Contribution

The study provides a comprehensive evaluation of ChatGPT's performance on code summarization, highlighting its current limitations compared to SOTA models.

Findings

ChatGPT performs worse than SOTA models on BLEU and ROUGE-L metrics.

ChatGPT has advantages and disadvantages in code summarization tasks.

The paper discusses open challenges and future opportunities for ChatGPT-based summarization.

Abstract

To support software developers in understanding and maintaining programs, various automatic code summarization techniques have been proposed to generate a concise natural language comment for a given code snippet. Recently, the emergence of large language models (LLMs) has led to a great boost in the performance of natural language processing tasks. Among them, ChatGPT is the most popular one which has attracted wide attention from the software engineering community. However, it still remains unclear how ChatGPT performs in (automatic) code summarization. Therefore, in this paper, we focus on evaluating ChatGPT on a widely-used Python dataset called CSN-Python and comparing it with several state-of-the-art (SOTA) code summarization models. Specifically, we first explore an appropriate prompt to guide ChatGPT to generate in-distribution comments. Then, we use such a prompt to ask ChatGPT to generate comments for all code snippets in the CSN-Python test set. We adopt three widely-used metrics (including BLEU, METEOR, and ROUGE-L) to measure the quality of the comments generated by ChatGPT and SOTA models (including NCS, CodeBERT, and CodeT5). The experimental results show that in terms of BLEU and ROUGE-L, ChatGPT's code summarization performance is significantly worse than all three SOTA models. We also present some cases and discuss the advantages and disadvantages of ChatGPT in code summarization. Based on the findings, we outline several open challenges and opportunities in ChatGPT-based code summarization.