How Robust are LLM-Generated Library Imports? An Empirical Study using Stack Overflow

TL;DR

This empirical study evaluates how well six state-of-the-art LLMs recommend Python libraries from Stack Overflow prompts, revealing preferences for third-party libraries, usability gaps, and areas for improving dependency handling.

Contribution

The paper provides a comprehensive analysis of LLMs' library import recommendations, highlighting their tendencies, limitations, and the need for better dependency management support.

Findings

LLMs favor third-party libraries over standard ones

4.6% of libraries could not be automatically resolved

Only two models provided installation guidance

Abstract

Software libraries are central to the functionality, security, and maintainability of modern code. As developers increasingly turn to Large Language Models (LLMs) to assist with programming tasks, understanding how these models recommend libraries is essential. In this paper, we conduct an empirical study of six state-of-the-art LLMs, both proprietary and open-source, by prompting them to solve real-world Python problems sourced from Stack Overflow. We analyze the types of libraries they import, the characteristics of those libraries, and the extent to which the recommendations are usable out of the box. Our results show that LLMs predominantly favour third-party libraries over standard ones, and often recommend mature, popular, and permissively licensed dependencies. However, we also identify gaps in usability: 4.6% of the libraries could not be resolved automatically due to structural mismatches between import names and installable packages, and only two models (out of six) provided installation guidance. While the generated code is technically valid, the lack of contextual support places the burden of manually resolving dependencies on the user. Our findings offer actionable insights for both developers and researchers, and highlight opportunities to improve the reliability and usability of LLM-generated code in the context of software dependencies.