Quantitative Risk Assessment in Radiation Oncology via LLM-Powered Root Cause Analysis of Incident Reports

TL;DR

This paper presents a novel LLM-based framework for automated root cause analysis and risk assessment in radiation oncology incident reports, enabling systematic identification of systemic vulnerabilities and predictors of event severity.

Contribution

It introduces a data-driven pipeline utilizing LLMs for structured incident classification and quantitative analysis, advancing risk assessment methods in radiation oncology.

Findings

Identified significant predictors of incident severity using regression models.

Uncovered systemic vulnerabilities through association rule mining.

Demonstrated the effectiveness of LLMs in automating root cause analysis.

Abstract

Background: Modern large language models (LLMs) offer powerful reasoning that converts narratives into structured, taxonomy-aligned data, revealing patterns across planning, delivery, and verification. Embedded as agentic tools, LLMs can assist root-cause analysis and risk assessment (e.g., failure mode and effect analysis FMEA), produce auditable rationales, and draft targeted mitigation actions. Methods: We developed a data-driven pipeline utilizing an LLM to perform automated root cause analysis on 254 institutional safety incidents. The LLM systematically classified each incident into structured taxonomies for radiotherapy pathway steps and contributory factors. Subsequent quantitative analyses included descriptive statistics, Analysis of Variance (ANOVA), multiple Ordinal Logistic Regression (OLR) analyses to identify predictors of event severity, and Association Rule Mining (ARM) to uncover systemic vulnerabilities. Results: The high-level Ordinal Logistic Regression (OLR) models identified specific, significant drivers of severity. The Pathway model was statistically significant (Pseudo R2 = 0.033, LR p = 0.015), as was the Responsibility model (Pseudo R2 = 0.028, LR p < 0.001). Association Rule Mining (ARM) identified high-confidence systemic rules, such as "CF5 Teamwork, management and organisational" (n = 8, Conf = 1.0) and the high-frequency link between "(11) Pre-treatment planning process" and "CF2 Procedural" (n = 152, Conf = 0.916). Conclusion: The LLM-powered, data-driven framework provides a more objective and powerful methodology for risk assessment than traditional approaches. Our findings empirically demonstrate that interventions focused on fortifying high-risk process steps and mitigating systemic failures are most effective for improving patient safety.