Repeatable Experiments with LabWiki

TL;DR

This paper introduces LabWiki, a workspace tool designed to facilitate the planning, execution, and documentation of experiments in distributed computing, ensuring repeatability and validation of scientific results.

Contribution

It presents a methodology and a set of tools that automate experiment documentation and support repeatability, building on existing standards and protocols.

Findings

LabWiki enables automatic capture of experiment dependencies.

The methodology improves experiment repeatability and validation.

Real experiments demonstrate the workspace's effectiveness.

Abstract

The ability to repeat the experiments from a research study and obtain similar results is a corner stone in experiment-based scientific discovery. This essential feature has been often ignored by the distributed computing and networking community. There are many reasons for that, such as the complexity of provisioning, configuring, and orchestrating the resources used by experiments, their multiple external dependencies, and the difficulty to seamlessly record these dependencies. This paper describes a methodology based on well-established principles to plan, prepare and execute experiments. We propose and describe a family of tools, the LabWiki workspace, to support an experimenter's workflow based on that methodology. This proposed workspace provides services and mechanisms for each step of an experiment-based study, while automatically capturing the necessary information to allow others to repeat, inspect, validate and modify prior experiments. Our LabWiki workspace builds on existing contributions, and de-facto protocol and model standards, which emerged from recent experimental facility initiatives. We use a real experiment as a thread to guide and illustrate the discussion throughout this paper.