Cross-study Reliability of the Open Card Sorting Method

TL;DR

This study empirically evaluates the cross-study reliability of open card sorting, demonstrating that it produces consistent content groupings and navigation schemes across different participant groups and studies.

Contribution

It provides empirical evidence supporting the reliability of open card sorting across multiple studies with different content types.

Findings

High similarity in card sorting data for the same content across studies

High agreement in the resulting navigation schemes

Supports the use of open card sorting for consistent information architecture

Abstract

Information architecture forms the foundation of users' navigation experience. Open card sorting is a widely-used method to create information architectures based on users' groupings of the content. However, little is known about the method's cross-study reliability: Does it produce consistent content groupings for similar profile participants involved in different card sort studies? This paper presents an empirical evaluation of the method's cross-study reliability. Six card sorts involving 140 participants were conducted: three open sorts for a travel website, and three for an eshop. Results showed that participants provided highly similar card sorting data for the same content. A rather high agreement of the produced navigation schemes was also found. These findings provide support for the cross-study reliability of the open card sorting method.