What Have We Learned from OpenReview?

TL;DR

This paper analyzes data from OpenReview to understand the effectiveness of open peer review, providing insights that could improve the process of writing, reviewing, and accepting scientific papers.

Contribution

It offers a comprehensive analysis of OpenReview data, revealing insights into open peer review's impact and effectiveness in computer science conferences.

Findings

OpenReview data shows trends in review quality and transparency.

Public reviews influence citation and revision patterns.

Insights suggest improvements for peer review practices.

Abstract

Anonymous peer review is used by the great majority of computer science conferences. OpenReview is such a platform that aims to promote openness in peer review process. The paper, (meta) reviews, rebuttals, and final decisions are all released to public. We collect 5,527 submissions and their 16,853 reviews from the OpenReview platform. We also collect these submissions' citation data from Google Scholar and their non-peer-reviewed versions from arXiv.org. By acquiring deep insights into these data, we have several interesting findings that could help understand the effectiveness of the public-accessible double-blind peer review process. Our results can potentially help writing a paper, reviewing it, and deciding on its acceptance.