Co-Refine: AI-Powered Tool Supporting Qualitative Analysis

TL;DR

Co-Refine is an AI-powered tool that provides real-time feedback on coding consistency in qualitative analysis, addressing temporal drift issues during large dataset coding.

Contribution

It introduces a novel three-stage audit pipeline combining deterministic metrics and LLM verdicts to ensure reliable, real-time quality control in qualitative coding workflows.

Findings

Deterministic embedding metrics effectively measure coding consistency.

LLM verdicts grounded within a small score range improve reliability.

The system enables continuous, real-time detection of interpretative drift.

Abstract

Qualitative coding relies on a researcher's application of codes to textual data. As coding proceeds across large datasets, interpretations of codes often shift (temporal drift), reducing the credibility of the analysis. Existing Computer-Assisted Qualitative Data Analysis (CAQDAS) tools provide support for data management but offer no workflow for real-time detection of these drifts. We present Co-Refine, an AI-augmented qualitative coding platform that delivers continuous, grounded feedback on coding consistency without disrupting the researcher's workflow. The system employs a three-stage audit pipeline: Stage 1 computes deterministic embedding-based metrics for mathematical consistency; Stage 2 grounds LLM verdicts within $\pm0.15$ of the deterministic scores; and Stage 3 produces code definitions from previous patterns to create a deepening feedback loop. Co-Refine demonstrates that deterministic scoring can effectively constrain LLM outputs to produce reliable, real-time audit signals for qualitative analysis.