An Agentic AI System for Multi-Framework Communication Coding

TL;DR

This paper introduces MOSAIC, an agentic AI system that uses a multi-agent architecture to improve clinical communication coding across various frameworks and domains, achieving high accuracy and reliability.

Contribution

The study presents a novel multi-agent AI system, MOSAIC, built on LangGraph architecture, capable of multi-framework clinical communication coding with improved performance and adaptability.

Findings

MOSAIC achieved an F1 score of 0.928 on test data.

Performance was highest in Rheumatology (F1=0.962).

MOSAIC outperformed baseline models in coding accuracy.

Abstract

Clinical communication is central to patient outcomes, yet large-scale human annotation of patient-provider conversation remains labor-intensive, inconsistent, and difficult to scale. Existing approaches based on large language models typically rely on single-task models that lack adaptability, interpretability, and reliability, especially when applied across various communication frameworks and clinical domains. In this study, we developed a Multi-framework Structured Agentic AI system for Clinical Communication (MOSAIC), built on a LangGraph-based architecture that orchestrates four core agents, including a Plan Agent for codebook selection and workflow planning, an Update Agent for maintaining up-to-date retrieval databases, a set of Annotation Agents that applies codebook-guided retrieval-augmented generation (RAG) with dynamic few-shot prompting, and a Verification Agent that provides consistency checks and feedback. To evaluate performance, we compared MOSAIC outputs against gold-standard annotations created by trained human coders. We developed and evaluated MOSAIC using 26 gold standard annotated transcripts for training and 50 transcripts for testing, spanning rheumatology and OB/GYN domains. On the test set, MOSAIC achieved an overall F1 score of 0.928. Performance was highest in the Rheumatology subset (F1 = 0.962) and strongest for Patient Behavior (e.g., patients asking questions, expressing preferences, or showing assertiveness). Ablations revealed that MOSAIC outperforms baseline benchmarking.