# Quality assessment of large language model–generated prior authorization letters in nephrology

## TL;DR

This study evaluates how well ChatGPT-5 can create prior authorization letters for nephrology medications, finding that while generally accurate, there are issues with coding and citations.

## Contribution

The study introduces a systematic evaluation of AI-generated prior authorization letters in nephrology, highlighting their potential and limitations.

## Key findings

- ChatGPT-5 generated PA letters with strong clinical reasoning in 89.7% of cases.
- ICD-10 coding was correct in 79.3% of letters, with errors mainly in CKD staging.
- Most letters cited valid references, but some had inaccessible links or incorrect citations.

## Abstract

Prior authorization (PA) is a major source of administrative burden, treatment delay, and clinician burnout. Artificial intelligence (AI), particularly large language models (LLMs), is increasingly used to assist with clinical documentation, yet its reliability for payer-facing administrative tasks remains uncertain.

To evaluate the quality of PA letters drafted by ChatGPT-5 for commonly used medications requiring PA in nephrology. Quality was evaluated based on correctness and strength of clinical reasoning.

We created a single standardized prompt and applied it across 29 nephrology scenarios to generate PA letters. Each PA letter was reviewed against four criteria: 1) absence of false statements or hallucinations, 2) correctness of ICD-10 coding, 3) presence and validity of citations, and 4) clinical reasoning, rated on a 4-point Likert scale (illogical, weak, adequate and strong). FDA drug labels, KDIGO guidelines and related randomized controlled trials were used as reference standards.

Out of 29 letters, one letter (3.5%) contained false statements mentioning an irrelevant clinical trial. The ICD-10 diagnosis code was correct in 23 letters (79.3%), most errors were related to chronic kidney disease (CKD) staging or internal diagnostic inconsistencies. 27 letters (93.1%) cited valid references, with one letter citing an incorrect trial and another one citing a correct KDIGO guideline with inaccessible link. Twenty-six letters (89.7%) demonstrated strong clinical reasoning, supported by guideline-oriented or FDA label–aligned justification. The remaining 3 letters were rated as adequate reasoning. The main areas for improvement involved citing relevant references and emphasizing special considerations, for example Risk Evaluation and Mitigation Strategy (REMS) compliance for eculizumab.

ChatGPT-5 can generate clinically coherent PA drafts for nephrology medications, but limitations in coding precision and citation reliability persist. With appropriate oversight, AI-assisted documentation may reduce administrative burden while maintaining safety and accuracy.