Beyond Advocacy: A Design Space for Replication-Related Studies

TL;DR

This paper introduces a multi-dimensional design space framework to systematically plan and analyze replication studies across scientific disciplines, aiding both retrospective and prospective research design decisions.

Contribution

It presents a novel framework that characterizes replication design choices as a pairwise comparison problem with four dimensions and three comparison levels.

Findings

Framework enables systematic comparison of replication designs.

Supports retrospective analysis and prospective planning of replication studies.

Relates to existing approaches in describing replication scope.

Abstract

The importance of replication is often discussed and advocated -- not only in the domains of visualization and HCI, but in all scientific areas. When replicating a study, design decisions need to be made with regards which aspects of the original study will remain the same and which will be altered. We present a supporting multi-dimensional design space framework within which such decisions can be identified, categorized, compared and analyzed. The framework treats replication experimental design as a pairwise comparison problem, and represents the design by four practical dimensions defined by three comparison levels. The design space is therefore a framework that can be used for both retrospective characterization and prospective planning. We provide worked examples, and relate our framework to other attempts at describing the scope of replication studies.