Development and Benchmarking of a Blended Human-AI Qualitative Research Assistant

TL;DR

This paper introduces Muse, an AI-powered qualitative research system that assists researchers in theme identification and annotation, demonstrating promising reliability and potential to enhance qualitative analysis while addressing human limitations.

Contribution

The paper presents a novel interactive AI system for qualitative research, benchmarking its performance against human analysis and analyzing its error modes for future improvements.

Findings

Inter-rater reliability of Cohen's κ = 0.71 with human coders

Muse can identify themes with substantial agreement to humans

Error analysis reveals key failure modes and bias correction capabilities

Abstract

Qualitative research emphasizes constructing meaning through iterative engagement with textual data. Traditionally this human-driven process requires navigating coder fatigue and interpretative drift, thus posing challenges when scaling analysis to larger, more complex datasets. Computational approaches to augment qualitative research have been met with skepticism, partly due to their inability to replicate the nuance, context-awareness, and sophistication of human analysis. Large language models, however, present new opportunities to automate aspects of qualitative analysis while upholding rigor and research quality in important ways. To assess their benefits and limitations - and build trust among qualitative researchers - these approaches must be rigorously benchmarked against human-generated datasets. In this work, we benchmark Muse, an interactive, AI-powered qualitative research system that allows researchers to identify themes and annotate datasets, finding an inter-rater reliability between Muse and humans of Cohen's $\kappa$ = 0.71 for well-specified codes. We also conduct robust error analysis to identify failure mode, guide future improvements, and demonstrate the capacity to correct for human bias.