A study on Prompt Design, Advantages and Limitations of ChatGPT for Deep Learning Program Repair

TL;DR

This paper investigates ChatGPT's effectiveness in repairing deep learning programs, focusing on prompt design, dialogue facilitation, and evaluating its capabilities in debugging, refactoring, and detecting issues.

Contribution

It introduces prompt templates and analyzes ChatGPT's strengths and limitations in DL program repair, addressing a gap in automatic repair for non-explicit decision logic.

Findings

ChatGPT can detect code smells and refactor code effectively.

Prompt design significantly influences repair performance.

Dialogue can enhance ChatGPT's debugging and repair capabilities.

Abstract

ChatGPT has revolutionized many research and industrial fields. ChatGPT has shown great potential in software engineering to boost various traditional tasks such as program repair, code understanding, and code generation. However, whether automatic program repair (APR) applies to deep learning (DL) programs is still unknown. DL programs, whose decision logic is not explicitly encoded in the source code, have posed unique challenges to APR. While to repair DL programs, an APR approach needs to not only parse the source code syntactically but also needs to understand the code intention. With the best prior work, the performance of fault localization is still far less than satisfactory (only about 30\%). Therefore, in this paper, we explore ChatGPT's capability for DL program repair by asking three research questions. (1) Can ChatGPT debug DL programs effectively? (2) How can ChatGPT's repair performance be improved by prompting? (3) In which way can dialogue help facilitate the repair? On top of that, we categorize the common aspects useful for prompt design for DL program repair. Also, we propose various prompt templates to facilitate the performance and summarize the advantages and disadvantages of ChatGPT's abilities such as detecting bad code smell, code refactoring, and detecting API misuse/deprecation.