Has the Machine Learning Review Process Become More Arbitrary as the Field Has Grown? The NeurIPS 2021 Consistency Experiment

TL;DR

This study analyzes the consistency of the NeurIPS 2021 review process, revealing significant randomness and arbitrariness, especially as the conference becomes more selective, highlighting challenges in objectively evaluating research quality.

Contribution

It extends previous experiments by quantifying review randomness at scale, showing increased arbitrariness with higher selectivity, and discussing implications for research assessment.

Findings

23% of papers had conflicting reviews

Approximately 50% of accepted papers would change if reviews were rerun

Increased selectivity correlates with higher arbitrariness

Abstract

We present the NeurIPS 2021 consistency experiment, a larger-scale variant of the 2014 NeurIPS experiment in which 10% of conference submissions were reviewed by two independent committees to quantify the randomness in the review process. We observe that the two committees disagree on their accept/reject recommendations for 23% of the papers and that, consistent with the results from 2014, approximately half of the list of accepted papers would change if the review process were randomly rerun. Our analysis suggests that making the conference more selective would increase the arbitrariness of the process. Taken together with previous research, our results highlight the inherent difficulty of objectively measuring the quality of research, and suggest that authors should not be excessively discouraged by rejected work.