Exploring the Challenges and Opportunities of AI-assisted Codebase Generation
Philipp Eibl, Sadra Sabouri, Souti Chattopadhyay

TL;DR
This study investigates how developers interact with AI-assisted codebase generators, identifying key challenges and barriers to improve their design and adoption based on user feedback and a survey of existing tools.
Contribution
The paper provides an empirical analysis of developer interactions with CBAs, highlights core dissatisfaction factors, and offers design insights to enhance future AI codebase assistants.
Findings
Participants often varied their prompts based on problem and structure.
Overall satisfaction with generated codebases was low (mean=2.8/5).
Functionality, code quality, and communication issues were primary dissatisfaction sources.
Abstract
Recent AI code assistants have significantly improved their ability to process more complex contexts and generate entire codebases based on a textual description, compared to the popular snippet-level generation. These codebase AI assistants (CBAs) can also extend or adapt codebases, allowing users to focus on higher-level design and deployment decisions. While prior work has extensively studied the impact of snippet-level code generation, this new class of codebase generation models is relatively unexplored. Despite initial anecdotal reports of excitement about these agents, they remain less frequently adopted compared to snippet-level code assistants. To utilize CBAs better, we need to understand how developers interact with CBAs, and how and why CBAs fall short of developers' needs. In this paper, we explored these gaps through a counterbalanced user study and interview with (n = 16)…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsSoftware Engineering Research · Teaching and Learning Programming · Artificial Intelligence in Healthcare and Education
