Scaling up the think-aloud method

TL;DR

This paper introduces automated transcription and analysis methods for the think-aloud technique, enabling large-scale cognitive research by processing verbal reasoning data efficiently using NLP tools.

Contribution

It develops automated methods for transcription and coding of think-aloud data, facilitating large-scale analysis of reasoning processes.

Findings

Automated transcription achieved moderate reliability with human coding.

Large sample analysis revealed patterns in reasoning traces.

Demonstrated feasibility of large-scale think-aloud data analysis.

Abstract

The think-aloud method, where participants voice their thoughts as they solve a task, is a valuable source of rich data about human reasoning processes. Yet, it has declined in popularity in contemporary cognitive science, largely because labor-intensive transcription and annotation preclude large sample sizes. Here, we develop methods to automate the transcription and annotation of verbal reports of reasoning using natural language processing tools, allowing for large-scale analysis of think-aloud data. In our study, 640 participants thought aloud while playing the Game of 24, a mathematical reasoning task. We automatically transcribed the recordings and coded the transcripts as search graphs, finding moderate inter-rater reliability with humans. We analyze these graphs and characterize consistency and variation in human reasoning traces. Our work demonstrates the value of think-aloud data at scale and serves as a proof of concept for the automated analysis of verbal reports.