Reflections on Traceability for Visualization Research

TL;DR

This paper explores how traceability can enhance transparency in design-oriented visualization research by documenting artifacts and processes, supported by a custom tool called tRRRacer.

Contribution

It introduces a theoretical framework for traceability in visualization research and presents a tool to operationalize its three core tenets.

Findings

The tRRRacer tool supports recording, reporting, and reading research artifacts.

Traceability enhances transparency in subjective, iterative design research.

Reflections suggest traceability can improve rigor without traditional reproducibility.

Abstract

Decades of advocacy for reproducibility and replication have advanced open, transparent practices in the sciences. However, traditional notions of reproducibility fit poorly with design-oriented visualization research, where insights emerge through subjective, situated, and iterative work. So how can we ensure rigor and transparency in processes that are inherently unreproducible? To introduce transparency in design-oriented research, we propose to focus on traceability: surfacing the origin and development of research contributions based on rich sets of artifacts documenting the design process. We investigated traceability through a collaborative autoethnographic reflection that builds on several years of work exploring ways to make design-oriented research transparent. This exploration includes an experiment to build a tool to support traceability, which we called tRRRacer. The tRRRacer tool provided a testbed for us to operationalize the three tenets of a traceable process: (1) Record abundant, annotated artifacts representative of research activities; (2) Report curated research threads that articulate rationale and evolution of the process, allowing others to (3) Read via interfaces that help retrace claims and assess plausibility. Reflecting on our experiences, we contribute a theorization of traceability and reflections on how we might support it.