What Makes a Scientific Paper be Accepted for Publication?

TL;DR

This paper explores how machine learning and explainable AI can analyze peer review processes, revealing factors influencing paper acceptance and providing causal explanations for review decisions.

Contribution

It introduces a methodology using linguistic features and confounding effect detection to generate causal explanations of peer review outcomes.

Findings

Reviewers' recommendations largely follow committee decisions

Originality, clarity, and substance are key acceptance factors

The approach offers insights into peer review transparency and bias

Abstract

Despite peer-reviewing being an essential component of academia since the 1600s, it has repeatedly received criticisms for lack of transparency and consistency. We posit that recent work in machine learning and explainable AI provide tools that enable insights into the decisions from a given peer review process. We start by extracting global explanations in the form of linguistic features that affect the acceptance of a scientific paper for publication on an open peer-review dataset. Second, since such global explanations do not justify causal interpretations, we provide a methodology for detecting confounding effects in natural language in order to generate causal explanations, under assumptions, in the form of lexicons. Our proposed linguistic explanation methodology indicates the following on a case dataset of ICLR submissions: a) the organising committee follows, for the most part, the recommendations of reviewers, and, b) the paper's main characteristics that led to reviewers recommending acceptance for publication are originality, clarity and substance.