Toward Human Centered Interactive Clinical Question Answering System

TL;DR

This paper presents an interactive clinical question answering system that leverages large language models to provide traceable, contextually aligned answers from unstructured clinical notes, improving usability for healthcare professionals.

Contribution

Introduces a novel human-centered interactive QA system for clinical notes using LLMs with zero-shot prompting, emphasizing transparency and usability in clinical workflows.

Findings

Semantic similarity scores exceeded 87%, indicating strong contextual understanding.

Exact string match scores ranged from 47 to 62%, showing moderate precision.

User evaluations highlighted strengths in design and answer accessibility.

Abstract

Unstructured clinical notes contain essential patient information but are challenging for physicians to search and interpret efficiently. Although large language models (LLMs) have shown promise in question answering (QA), most existing systems lack transparency, usability, and alignment with clinical workflows. This work introduces an interactive QA system that enables physicians to query clinical notes via text or voice and receive extractive answers highlighted directly in the note for traceability. The system was built using OpenAI models with zero-shot prompting and evaluated across multiple metrics, including exact string match, word overlap, SentenceTransformer similarity, and BERTScore. Results show that while exact match scores ranged from 47 to 62 percent, semantic similarity scores exceeded 87 percent, indicating strong contextual alignment even when wording varied. To assess usability, the system was also evaluated using simulated clinical personas. Seven diverse physician and nurse personas interacted with the system across scenario-based tasks and provided structured feedback. The evaluations highlighted strengths in intuitive design and answer accessibility, alongside opportunities for enhancing explanation clarity.