An empirical approach to interpreting card-sorting data

TL;DR

This paper introduces a novel microscopic graph-theoretic approach to analyze card-sorting data, revealing that individual sorting styles like 'stacker' versus 'spreader' explain more variation than expertise level.

Contribution

It presents a new graph-based analysis method for card-sorting data that uncovers individual sorting styles beyond expert-novice distinctions.

Findings

Most variation in sorting is due to individual styles, not expertise.

The 'stacker' versus 'spreader' characteristic explains significant differences.

Traditional descriptive statistics often miss these individual differences.

Abstract

Since it was first published 30 years ago, Chi et al.'s seminal paper on expert and novice categorization of introductory problems led to a plethora of follow-up studies within and outside of the area of physics [Chi et al. Cognitive Science 5, 121 - 152 (1981)]. These studies frequently encompass "card-sorting" exercises whereby the participants group problems. While this technique certainly allows insights into problem solving approaches, simple descriptive statistics more often than not fail to find significant differences between experts and novices. In moving beyond descriptive statistics, we describe a novel microscopic approach that takes into account the individual identity of the cards and uses graph theory and models to visualize, analyze, and interpreting problem categorization experiments. We apply these methods to an introductory physics (mechanics) problem categorization experiment, and find that most of the variation in sorting outcome is not due to the sorter being an expert versus a novice, but rather due to an independent characteristic that we named "stacker" versus "spreader." The fact that the expert-novice distinction only accounts for a smaller amount of the variation may explain the frequent null-results when conducting these experiments.