Evaluating and Improving ChatGPT-Based Expansion of Abbreviations

TL;DR

This paper empirically evaluates ChatGPT for expanding abbreviations in source code, identifies its limitations, and proposes techniques to improve its accuracy, making it comparable to specialized methods without complex parsing.

Contribution

First empirical study on LLM-based abbreviation expansion in source code, analyzing failure causes, and proposing context selection, iterative abbreviation recognition, and post-checking techniques.

Findings

ChatGPT is less accurate than state-of-the-art methods by ~28%.

Context selection improves abbreviation recognition.

Post-condition checks reduce incorrect expansions.

Abstract

Source code identifiers often contain abbreviations. Such abbreviations may reduce the readability of the source code, which in turn hinders the maintenance of the software applications. To this end, accurate and automated approaches to expanding abbreviations in source code are desirable and abbreviation expansion has been intensively investigated. However, to the best of our knowledge, most existing approaches are heuristics, and none of them has even employed deep learning techniques, let alone the most advanced large language models (LLMs). LLMs have demonstrated cutting-edge performance in various software engineering tasks, and thus it has the potential to expand abbreviation automatically. To this end, in this paper, we present the first empirical study on LLM-based abbreviation expansion. Our evaluation results on a public benchmark suggest that ChatGPT is substantially less accurate than the state-of-the-art approach, reducing precision and recall by 28.2\% and 27.8\%, respectively. We manually analyzed the failed cases, and discovered the root causes for the failures: 1) Lack of contexts and 2) Inability to recognize abbreviations. In response to the first cause, we investigated the effect of various contexts and found surrounding source code is the best selection. In response to the second cause, we designed an iterative approach that identifies and explicitly marks missed abbreviations in prompts. Finally, we proposed a post-condition checking to exclude incorrect expansions that violate commonsense. All such measures together make ChatGPT-based abbreviation expansion comparable to the state of the art while avoiding expensive source code parsing and deep analysis that are indispensable for state-of-the-art approaches.