ConvScale: Conversational Interviews for Scale-Aligned Measurement

TL;DR

ConvScale introduces an AI-supported method to transform psychometric scales into conversational interviews, aiming to enhance quantitative measurement by predicting item scores from natural dialogue, with promising alignment to self-reports.

Contribution

This paper presents ConvScale, a novel approach that converts psychometric scales into conversational interviews and predicts quantitative scores from dialogue data.

Findings

Scores predicted by ConvScale closely match self-reports at item and construct levels

Moderate internal reliability achieved in the conversational assessment

Structural validity of the conversational approach was found to be inadequate

Abstract

Conversational interviews are commonly used to complement structured surveys by eliciting rich and contextualized responses, which are typically analyzed qualitatively. However, their potential contribution to quantitative measurement remains underexplored. In this paper, we introduce ConvScale, an AI-supported approach that transforms psychometric scales into natural conversational interviews while preserving the original measurement structure. Based on interview data, ConvScale predicts item-level scores and aggregates them to derive scale-based assessments. In a within-subjects study with 18 participants, our results show that ConvScale-derived scores align closely with participants' self-report scores at both the item and construct levels, while maintaining moderate internal reliability; however, the structural validity was inadequate. In light of this, we discussed the potential of supporting quantitative measurement through interviews and proposed implications for future designs.