Is ChatGPT the Ultimate Programming Assistant -- How far is it?
Haoye Tian, Weiqi Lu, Tsz On Li, Xunzhu Tang, Shing-Chi Cheung,, Jacques Klein, Tegawend\'e F. Bissyand\'e

TL;DR
This paper empirically evaluates ChatGPT's capabilities as an automated programming assistant, focusing on code generation, repair, and summarization, revealing both its strengths and limitations in practical software engineering tasks.
Contribution
It provides a comprehensive empirical analysis of ChatGPT's performance on unseen programming problems and compares it with state-of-the-art methods, highlighting its potential and constraints.
Findings
Effective in handling common programming problems
Limitations due to attention span and prompt complexity
Ability to reason about original code intentions
Abstract
Recently, the ChatGPT LLM has received great attention: it can be used as a bot for discussing source code, prompting it to suggest changes, provide descriptions or even generate code. Typical demonstrations generally focus on existing benchmarks, which may have been used in model training (i.e., data leakage). To assess the feasibility of using an LLM as a useful assistant bot for programmers, we must assess its realistic capabilities on unseen problems as well as its capabilities on various tasks. In this paper, we present an empirical study of ChatGPT's potential as a fully automated programming assistant, focusing on the tasks of code generation, program repair, and code summariziation. The study investigates ChatGPT's performance on common programming problems and compares it with state-of-the-art approaches on two benchmarks. Among several findings, our study shows that ChatGPT is…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsSoftware Engineering Research · Software Reliability and Analysis Research · Artificial Intelligence in Healthcare and Education
