Textual analysis of artificial intelligence manuscripts reveals features associated with peer review outcome

TL;DR

This study analyzes AI manuscripts to identify linguistic and semantic features that correlate with peer review outcomes, revealing differences in readability, jargon, word usage, and citation patterns between accepted and rejected papers.

Contribution

It introduces a comprehensive linguistic and semantic analysis approach to predict peer review outcomes based on manuscript features, highlighting specific language patterns associated with acceptance.

Findings

Accepted papers used more scientific jargon and less readable language.

Accepted papers cited fewer unique references and had higher semantic similarity.

Words related to machine learning and neural networks increased acceptance likelihood.

Abstract

We analysed a dataset of scientific manuscripts that were submitted to various conferences in artificial intelligence. We performed a combination of semantic, lexical and psycholinguistic analyses of the full text of the manuscripts and compared them with the outcome of the peer review process. We found that accepted manuscripts scored lower than rejected manuscripts on two indicators of readability, and that they also used more scientific and artificial intelligence jargon. We also found that accepted manuscripts were written with words that are less frequent, that are acquired at an older age, and that are more abstract than rejected manuscripts. The analysis of references included in the manuscripts revealed that the subset of accepted submissions were more likely to cite the same publications. This finding was echoed by pairwise comparisons of the word content of the manuscripts (i.e. an indicator or semantic similarity), which were more similar in the subset of accepted manuscripts. Finally, we predicted the peer review outcome of manuscripts with their word content, with words related to machine learning and neural networks positively related with acceptance, whereas words related to logic, symbolic processing and knowledge-based systems negatively related with acceptance.