Card Sorting Simulator: Augmenting Design of Logical Information Architectures with Large Language Models

TL;DR

This paper introduces a prototype AI-powered tool using Large Language Models to simulate card sorting, aiming to provide automated feedback and improve the design of information architectures, validated through a comprehensive comparison with real user data.

Contribution

It presents the Card Sorting Simulator, the first tool leveraging LLMs for automated card sorting simulation, validated against extensive real-world data.

Findings

Good agreement between simulated and real clusterings

Inconsistencies due to mental model differences

Complexity of cards affects simulation accuracy

Abstract

Card sorting is a common ideation technique that elicits information on users' mental organization of content and functionality by having them sort items into categories. For more robust card sorting research, digital card sorting tools could benefit from providing quick automated feedback. Our objective of this research is to advance toward an instrument that applies artificial intelligence (AI) to augment card sorting. For this purpose, we develop the Card Sorting Simulator, a prototype tool that leverages Large Language Models (LLMs) to generate informative categorizations of cards. To illuminate how aligned the simulation is with card sorting by actual participants, and to inform the instrument's design decisions, we conducted a generalizability-focused comparative study. We obtained 28 pre-existing card sorting studies from real practitioners, comprising 1,399 participants, along with diverse contents and origins. With this dataset, we conducted a comprehensive and nuanced analysis of the agreement between actual card sorting results (clusterings of cards) and synthetic clusterings across a multitude of LLMs and prompt designs. Mutual information scores indicate a good degree of agreement to real result clustering, although similarity matrices also demonstrate inconsistencies from mental models, which can be attributed to their top-down nature. Furthermore, the number of cards or complexity of their labels impact the accuracy of its simulation. These findings bolster the case for AI augmentation in card sorting research as a source of meaningful preliminary feedback and highlight the need for further study for the development and validation of intelligent user research tools.