A Dataset and Preliminary Study of Using GPT-5 for Code-change Impact Analysis

TL;DR

This study evaluates GPT-5's ability to predict impacted code entities from source code changes, revealing limited performance but some improvement with additional diff information, highlighting challenges in applying LLMs to impact analysis.

Contribution

First comprehensive assessment of GPT-5 for code-change impact analysis, introducing a new dataset and exploring different information configurations.

Findings

GPT-5 outperforms GPT-5-mini in impact prediction.

Adding diff hunks slightly improves model performance.

Both models perform poorly overall in the task.

Abstract

Understanding source code changes and their impact on other code entities is a crucial skill in software development. However, the analysis of code changes and their impact is often performed manually and therefore is time-consuming. Recent advancements in AI, and in particular large language models (LLMs) show promises to help developers in various code analysis tasks. However, the extent to which this potential can be utilized for understanding code changes and their impact is underexplored. To address this gap, we study the capabilities of GPT-5 and GPT-5-mini to predict the code entities impacted by given source code changes. We construct a dataset containing information about seed-changes, change pairs, and change types for each commit. Existing datasets lack crucial information about seed changes and impacted code entities. Our experiments evaluate the LLMs in two configurations: (1) seed-change information and the parent commit tree and (2) seed-change information, the parent commit tree, and the diff hunk of each seed change. We found that both LLMs perform poorly in the two experiments, whereas GPT-5 outperforms GPT-5-mini. Furthermore, the provision of the diff hunks helps both models to slightly improve their performance.