RelevAI-Reviewer: A Benchmark on AI Reviewers for Survey Paper Relevance

TL;DR

RelevAI-Reviewer introduces an AI-based system and dataset for automating survey paper relevance assessment, aiming to streamline peer review and improve accuracy using machine learning models like BERT.

Contribution

The paper presents a novel dataset and formulates survey paper relevance assessment as a classification task, demonstrating the effectiveness of BERT-based models over traditional methods.

Findings

BERT-based classifiers outperform traditional ML methods.

A new dataset with 25,164 instances for relevance classification.

Preliminary results show improved accuracy with advanced language models.

Abstract

Recent advancements in Artificial Intelligence (AI), particularly the widespread adoption of Large Language Models (LLMs), have significantly enhanced text analysis capabilities. This technological evolution offers considerable promise for automating the review of scientific papers, a task traditionally managed through peer review by fellow researchers. Despite its critical role in maintaining research quality, the conventional peer-review process is often slow and subject to biases, potentially impeding the swift propagation of scientific knowledge. In this paper, we propose RelevAI-Reviewer, an automatic system that conceptualizes the task of survey paper review as a classification problem, aimed at assessing the relevance of a paper in relation to a specified prompt, analogous to a "call for papers". To address this, we introduce a novel dataset comprised of 25,164 instances. Each instance contains one prompt and four candidate papers, each varying in relevance to the prompt. The objective is to develop a machine learning (ML) model capable of determining the relevance of each paper and identifying the most pertinent one. We explore various baseline approaches, including traditional ML classifiers like Support Vector Machine (SVM) and advanced language models such as BERT. Preliminary findings indicate that the BERT-based end-to-end classifier surpasses other conventional ML methods in performance. We present this problem as a public challenge to foster engagement and interest in this area of research.