Editing with AI: How Doctors Refine LLM-Generated Answers to Patient Queries

TL;DR

This study explores how ophthalmologists refine AI-generated responses to patient questions, highlighting the importance of contextualization, human oversight, and workflow trade-offs in clinical communication.

Contribution

It provides the first detailed analysis of physician editing behaviors and workflows when using LLMs for patient communication in ophthalmology.

Findings

Contextualization is the main editing activity.

Indirect editing reduces effort but may introduce errors.

Direct editing ensures accuracy but increases workload.

Abstract

Patients frequently seek information during their medical journeys, but the rising volume of digital patient messages has strained healthcare systems. Large language models (LLMs) offer promise in generating draft responses for clinicians, yet how physicians refine these drafts remains underexplored. We present a mixed-methods study with nine ophthalmologists answering 144 cataract surgery questions across three conditions: writing from scratch, directly editing LLM drafts, and instruction-based indirect editing. Our quantitative and qualitative analyses reveal that while LLM outputs were generally accurate, occasional errors and automation bias revealed the need for human oversight. Contextualization--adapting generic answers to local practices and patient expectations--emerged as a dominant form of editing. Editing workflows revealed trade-offs: indirect editing reduced effort but introduced errors, while direct editing ensured precision but with higher workload. We conclude with design and policy implications for building safe, scalable LLM-assisted clinical communication systems.