Artifact Evaluation for Distributed Systems: Current Practices and Beyond

TL;DR

This paper reviews current artifact evaluation practices in distributed systems research, highlighting gaps and proposing recommendations to improve artifact quality and reproducibility in the community.

Contribution

It identifies shortcomings in current artifact evaluation procedures for distributed systems and offers checklists and guidelines to enhance their effectiveness and consistency.

Findings

Artifact evaluation in distributed systems is less unified than in other fields.

Current assessment criteria are inadequate for distributed systems' complexities.

Recommendations aim to improve artifact submission quality and reproducibility.

Abstract

Although repeatability and reproducibility are essential in science, failed attempts to replicate results across diverse fields made some scientists argue for a reproducibility crisis. In response, several high-profile venues within computing established artifact evaluation tracks, a systematic procedure for evaluating and badging research artifacts, with an increasing number of artifacts submitted. This study compiles recent artifact evaluation procedures and guidelines to show how artifact evaluation in distributed systems research lags behind other computing disciplines and/or is less unified and more complex. We further argue that current artifact assessment criteria are uncoordinated and insufficient for the unique challenges of distributed systems research. We examine the current state of the practice for artifacts and their evaluation to provide recommendations to assist artifact authors, reviewers, and track chairs. We summarize the recommendations and best practices as checklists for artifact authors and evaluation committees. Although our recommendations alone will not resolve the repeatability and reproducibility crisis, we want to start a discussion in our community to increase the number of submitted artifacts and their quality over time.