Standardization of Post-Publication Code Verification by Journals is Possible with the Support of the Community

TL;DR

This paper proposes a community-supported framework for journals to implement formal post-publication code verification, enhancing reproducibility in machine learning research through verification badges linked to code repositories.

Contribution

It introduces a modified verification badge system allowing independent post-publication code replication submissions, formalizing verification in academic publishing.

Findings

Verification badges increase transparency and reproducibility.

Community involvement can support formal post-publication verification.

Framework is adaptable to existing publication practices.

Abstract

Reproducibility remains a challenge in machine learning research. While code and data availability requirements have become increasingly common, post-publication verification in journals is still limited and unformalized. This position paper argues that it is plausible for journals and conference proceedings to implement post-publication verification. We propose a modification to ACM pre-publication verification badges that allows independent researchers to submit post-publication code replications to the journal, leading to visible verification badges included in the article metadata. Each article may earn up to two badges, each linked to verified code in its corresponding public repository. We describe the motivation, related initiatives, a formal framework, the potential impact, possible limitations, and alternative views.