Efficiency with Rigor! A Trustworthy LLM-powered Workflow for Qualitative Data Analysis

TL;DR

This paper introduces MindCoder, a transparent, LLM-powered workflow for qualitative data analysis that enhances trustworthiness by combining automation with human interpretive control and detailed logging.

Contribution

It presents a novel workflow, MindCoder, that balances automation and human involvement in QDA, ensuring transparency and trustworthiness.

Findings

MindCoder supports active interpretation and flexible control.

It produces more trustworthy codebooks.

The workflow maintains comprehensive logs for transparency.

Abstract

Qualitative data analysis (QDA) emphasizes trustworthiness, requiring sustained human engagement and reflexivity. Recently, large language models (LLMs) have been applied in QDA to improve efficiency. However, their use raises concerns about unvalidated automation and displaced sensemaking, which can undermine trustworthiness. To address these issues, we employed two strategies: transparency and human involvement. Through a literature review and formative interviews, we identified six design requirements for transparent automation and meaningful human involvement. Guided by these requirements, we developed MindCoder, an LLM-powered workflow that delegates mechanical tasks, such as grouping and validation, to the system, while enabling humans to conduct meaningful interpretation. MindCoder also maintains comprehensive logs of users' step-by-step interactions to ensure transparency and support trustworthy results. In an evaluation with 12 users and two external evaluators, MindCoder supported active interpretation, offered flexible control, and produced more trustworthy codebooks. We further discuss design implications for building human-AI collaborative QDA workflows.