Is ChatGPT a Good Software Librarian? An Exploratory Study on the Use of ChatGPT for Software Library Recommendations
Jasmine Latendresse, SayedHassan Khatoonabadi, Ahmad Abdellatif, Emad, Shihab

TL;DR
This study evaluates ChatGPT's effectiveness in recommending Python libraries for developers, revealing strengths in recommending popular options but also highlighting issues with licensing clarity and library usability.
Contribution
The paper provides an empirical assessment of ChatGPT's library recommendation capabilities, identifying key areas for improvement in licensing transparency and usability.
Findings
ChatGPT recommends third-party libraries 10% more often than humans.
14.2% of recommended libraries have restrictive licenses.
6.5% of libraries do not work out of the box.
Abstract
Software libraries play a critical role in the functionality, efficiency, and maintainability of software systems. As developers increasingly rely on Large Language Models (LLMs) to streamline their coding processes, the effectiveness of these models in recommending appropriate libraries becomes crucial yet remains largely unexplored. In this paper, we assess the effectiveness of ChatGPT as a software librarian and identify areas for improvement. We conducted an empirical study using GPT-3.5 Turbo to generate Python code for 10,000 Stack Overflow questions. Our findings show that ChatGPT uses third-party libraries nearly 10% more often than human developers, favoring widely adopted and well-established options. However, 14.2% of the recommended libraries had restrictive copyleft licenses, which were not explicitly communicated by ChatGPT. Additionally, 6.5% of the libraries did not work…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsArtificial Intelligence in Healthcare and Education · FinTech, Crowdfunding, Digital Finance · Online Learning and Analytics
