The Foundation Cracks: A Comprehensive Study on Bugs and Testing Practices in LLM Libraries
Weipeng Jiang, Xiaoyu Zhang, Xiaofei Xie, Jiongchi Yu, Yuhan Zhi, Shiqing Ma, Chao Shen

TL;DR
This study provides a detailed empirical analysis of bugs and testing practices in LLM libraries, revealing API misuse as the main root cause and highlighting gaps in current testing effectiveness.
Contribution
It offers the first comprehensive taxonomy of bug types and root causes in LLM libraries, along with an analysis of testing strategies and their limitations.
Findings
API misuse is the leading root cause of bugs.
Most bugs escape detection due to inadequate testing.
Predefined expected outputs are the most common testing strategy.
Abstract
Large Language Model (LLM) libraries have emerged as the foundational infrastructure powering today's AI revolution, serving as the backbone for LLM deployment, inference optimization, fine-tuning, and production serving across diverse applications. Despite their critical role in the LLM ecosystem, these libraries face frequent quality issues and bugs that threaten the reliability of AI systems built upon them. To address this knowledge gap, we present the first comprehensive empirical investigation into bug characteristics and testing practices in modern LLM libraries. We examine 313 bug-fixing commits extracted across two widely-adopted LLM libraries: HuggingFace Transformers and vLLM.Through rigorous manual analysis, we establish comprehensive taxonomies categorizing bug symptoms into 5 types and root causes into 14 distinct categories.Our primary discovery shows that API misuse has…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsDigital Rights Management and Security
