Update on the Systematics in the ALMA Proposal Review Process after Cycle 8

TL;DR

This paper analyzes recent changes in ALMA proposal review systematics after implementing dual-anonymous review and distributed peer review, highlighting biases related to PI experience and regional affiliation.

Contribution

It provides an updated analysis of review biases and systematics in ALMA proposal rankings after significant process changes in Cycles 7 and 8.

Findings

Experienced PIs tend to have better ranks.

Second-time submitters show improved ranks.

Regional biases persist in proposal rankings.

Abstract

We present an updated analysis of systematics in the Atacama Large Millimeter/submillimeter Array (ALMA) proposal ranks from Carpenter (2020) to include the last two ALMA cycles, when significant changes were introduced in the proposal review process. In Cycle 7, the investigator list on the proposal cover sheet was randomized such that the reviewers were aware of the overall proposal team but did not know the identity of the principal investigator (PI). In Cycle 8, ALMA adopted distributed peer review for most proposals and implemented dual-anonymous review for all proposals, in which the identity of the proposal team was not revealed to the reviewers. The most significant change in the systematics in Cycles 7 and 8 compared to previous cycles is related to the experience of PIs in submitting ALMA proposals. PIs that submit a proposal every cycle tend to have ranks that are consistent with average in Cycles 7 and 8 whereas previously they had the best overall ranks. Also, PIs who submitted a proposal for the second time show improved ranks over previous cycles. These results suggest some biases related to the relative prominence of the PI have been present in the ALMA review process. Systematics related to regional affiliation remain largely unchanged in that PIs from Chile, East Asia, and non-ALMA regions tend to have poorer overall ranks than PIs from Europe and North America. The systematics of how one region ranks proposals from another region are also investigated. No significant differences in the overall ranks based on gender of the PI are observed.