A Multi-Agent Large Language Model Framework for Automated Qualitative Analysis

TL;DR

This paper introduces CoTI, a multi-agent LLM framework that automates qualitative thematic analysis of patient interviews, matching expert-level insights and enabling AI-human collaboration in healthcare research.

Contribution

The study presents a novel multi-agent LLM framework, CoTI, for automating qualitative analysis and demonstrates its effectiveness in healthcare interview analysis.

Findings

CoTI identified themes similar to senior investigators.

CoTI outperformed baseline NLP models.

AI-human collaboration showed limited additional benefit.

Abstract

Understanding patients experiences is essential for advancing patient centered care, especially in chronic diseases that require ongoing communication. However, qualitative thematic analysis, the primary approach for exploring these experiences, remains labor intensive, subjective, and difficult to scale. In this study, we developed a multi agent large language model framework that automates qualitative thematic analysis through three agents (Instructor, Thematizer, CodebookGenerator), named Collaborative Theme Identification Agent (CoTI). We applied CoTI to 12 heart failure patient interviews to analyze their perceptions of medication intensity. CoTI identified key phrases, themes, and codebook that were more similar to those of the senior investigator than both junior investigators and baseline NLP models. We also implemented CoTI into a user-facing application to enable AI human interaction in qualitative analysis. However, collaboration between CoTI and junior investigators provided only marginal gains, suggesting they may overrely on CoTI and limit their independent critical thinking.