Unveiling the Sentinels: Assessing AI Performance in Cybersecurity Peer Review

TL;DR

This study evaluates AI models' ability to predict peer review outcomes in cybersecurity research, comparing machine learning approaches and human reviewers, highlighting AI's potential and limitations in automating parts of the review process.

Contribution

The paper introduces a comprehensive dataset and compares ChatGPT with Doc2Vec-based models for predicting review results, demonstrating over 90% accuracy with the latter.

Findings

Doc2Vec-based models outperform ChatGPT in prediction accuracy

AI models can assist in review outcome prediction with high reliability

Human judgment remains essential in certain review aspects

Abstract

Peer review is the method employed by the scientific community for evaluating research advancements. In the field of cybersecurity, the practice of double-blind peer review is the de-facto standard. This paper touches on the holy grail of peer reviewing and aims to shed light on the performance of AI in reviewing for academic security conferences. Specifically, we investigate the predictability of reviewing outcomes by comparing the results obtained from human reviewers and machine-learning models. To facilitate our study, we construct a comprehensive dataset by collecting thousands of papers from renowned computer science conferences and the arXiv preprint website. Based on the collected data, we evaluate the prediction capabilities of ChatGPT and a two-stage classification approach based on the Doc2Vec model with various classifiers. Our experimental evaluation of review outcome prediction using the Doc2Vec-based approach performs significantly better than the ChatGPT and achieves an accuracy of over 90%. While analyzing the experimental results, we identify the potential advantages and limitations of the tested ML models. We explore areas within the paper-reviewing process that can benefit from automated support approaches, while also recognizing the irreplaceable role of human intellect in certain aspects that cannot be matched by state-of-the-art AI techniques.