An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models

TL;DR

This paper empirically analyzes outages and failure-recovery patterns in public large language model services, revealing key differences and periodicities to inform better system design and usage.

Contribution

It provides the first comprehensive empirical characterization of outages and failure-recovery in major public LLM services, with detailed statistical analysis and publicly available datasets.

Findings

OpenAI's ChatGPT failures are less frequent but take longer to resolve.

Service failures show strong weekly and monthly periodicity.

OpenAI services have better failure-isolation than Anthropic services.

Abstract

People and businesses increasingly rely on public LLM services, such as ChatGPT, DALLE, and Claude. Understanding their outages, and particularly measuring their failure-recovery processes, is becoming a stringent problem. However, only limited studies exist in this emerging area. Addressing this problem, in this work we conduct an empirical characterization of outages and failure-recovery in public LLM services. We collect and prepare datasets for 8 commonly used LLM services across 3 major LLM providers, including market-leads OpenAI and Anthropic. We conduct a detailed analysis of failure recovery statistical properties, temporal patterns, co-occurrence, and the impact range of outage-causing incidents. We make over 10 observations, among which: (1) Failures in OpenAI's ChatGPT take longer to resolve but occur less frequently than those in Anthropic's Claude;(2) OpenAI and Anthropic service failures exhibit strong weekly and monthly periodicity; and (3) OpenAI services offer better failure-isolation than Anthropic services. Our research explains LLM failure characteristics and thus enables optimization in building and using LLM systems. FAIR data and code are publicly available on https://zenodo.org/records/14018219 and https://github.com/atlarge-research/llm-service-analysis.