Exploring Large Language Models for Analyzing and Improving Method Names in Scientific Code

TL;DR

This paper evaluates the effectiveness of four large language models in analyzing and suggesting improvements for method names in scientific Python code, highlighting their strengths and limitations.

Contribution

It introduces a systematic evaluation of LLMs for scientific code naming analysis, revealing their potential and current shortcomings in domain-specific contexts.

Findings

LLMs can identify grammatical patterns in method names

They tend to follow good naming practices like starting with verbs

Automated suggestions still need human review due to inconsistencies

Abstract

Research scientists increasingly rely on implementing software to support their research. While previous research has examined the impact of identifier names on program comprehension in traditional programming environments, limited work has explored this area in scientific software, especially regarding the quality of method names in the code. The recent advances in Large Language Models (LLMs) present new opportunities for automating code analysis tasks, such as identifier name appraisals and recommendations. Our study evaluates four popular LLMs on their ability to analyze grammatical patterns and suggest improvements for 496 method names extracted from Python-based Jupyter Notebooks. Our findings show that the LLMs are somewhat effective in analyzing these method names and generally follow good naming practices, like starting method names with verbs. However, their inconsistent handling of domain-specific terminology and only moderate agreement with human annotations indicate that automated suggestions require human evaluation. This work provides foundational insights for improving the quality of scientific code through AI automation.