A Guide to Computational Reproducibility in Signal Processing and Machine Learning

TL;DR

This paper discusses the challenges of computational reproducibility in signal processing and machine learning, highlighting the need for practical tools and standards to improve reproducibility practices among researchers.

Contribution

It provides a comprehensive overview of obstacles to reproducibility and offers practical tools and strategies tailored for signal processing and machine learning researchers.

Findings

Many experiments are difficult to reproduce due to new challenges.

Lack of clear standards hampers reproducibility efforts.

Proposed tools and strategies can improve reproducibility.

Abstract

Computational reproducibility is a growing problem that has been extensively studied among computational researchers and within the signal processing and machine learning research community. However, with the changing landscape of signal processing and machine learning research come new obstacles and unseen challenges in creating reproducible experiments. Due to these new challenges most computational experiments have become difficult, if not impossible, to be reproduced by an independent researcher. In 2016 a survey conducted by the journal Nature found that 50% of researchers were unable to reproduce their own experiments. While the issue of computational reproducibility has been discussed in the literature and specifically within the signal processing community, it is still unclear to most researchers what are the best practices to ensure reproducibility without impinging on their primary responsibility of conducting research. We feel that although researchers understand the importance of making experiments reproducible, the lack of a clear set of standards and tools makes it difficult to incorporate good reproducibility practices in most labs. It is in this regard that we aim to present signal processing researchers with a set of practical tools and strategies that can help mitigate many of the obstacles to producing reproducible computational experiments.