Why Does the LLM Stop Computing: An Empirical Study of User-Reported Failures in Open-Source LLMs

TL;DR

This study empirically analyzes 705 real-world failures in open-source LLMs, revealing systemic fragility in deployment stacks and providing insights to improve reliability in user-managed environments.

Contribution

It is the first large-scale empirical investigation into open-source LLM failures, highlighting systemic issues in deployment and offering actionable guidance for robustness.

Findings

Runtime crashes indicate infrastructure issues

Incorrect outputs often stem from tokenizer defects

Reliability barriers are ecosystem-wide, not architecture-specific

Abstract

The democratization of open-source Large Language Models (LLMs) allows users to fine-tune and deploy models on local infrastructure but exposes them to a First Mile deployment landscape. Unlike black-box API consumption, the reliability of user-managed orchestration remains a critical blind spot. To bridge this gap, we conduct the first large-scale empirical study of 705 real-world failures from the open-source DeepSeek, Llama, and Qwen ecosystems. Our analysis reveals a paradigm shift: white-box orchestration relocates the reliability bottleneck from model algorithmic defects to the systemic fragility of the deployment stack. We identify three key phenomena: (1) Diagnostic Divergence: runtime crashes distinctively signal infrastructure friction, whereas incorrect functionality serves as a signature for internal tokenizer defects. (2) Systemic Homogeneity: Root causes converge across divergent series, confirming reliability barriers are inherent to the shared ecosystem rather than specific architectures. (3) Lifecycle Escalation: Barriers escalate from intrinsic configuration struggles during fine-tuning to compounded environmental incompatibilities during inference. Supported by our publicly available dataset, these insights provide actionable guidance for enhancing the reliability of the LLM landscape.