WarrantScore: Modeling Warrants between Claims and Evidence for Substantiation Evaluation in Peer Reviews

TL;DR

This paper introduces WarrantScore, a new metric that evaluates the logical inference between claims and evidence in peer reviews, aiming to improve the interpretability and accuracy of substantiation assessment in scientific review comments.

Contribution

It proposes a novel evaluation metric that assesses logical inference between claims and evidence, enhancing the interpretability and correlation with human judgments in peer review analysis.

Findings

Achieves higher correlation with human scores than conventional methods

Improves assessment of logical inference in scientific reviews

Supports more efficient peer-review processes

Abstract

The scientific peer-review process is facing a shortage of human resources due to the rapid growth in the number of submitted papers. The use of language models to reduce the human cost of peer review has been actively explored as a potential solution to this challenge. A method has been proposed to evaluate the level of substantiation in scientific reviews in a manner that is interpretable by humans. This method extracts the core components of an argument, claims and evidence, and assesses the level of substantiation based on the proportion of claims supported by evidence. The level of substantiation refers to the extent to which claims are based on objective facts. However, when assessing the level of substantiation, simply detecting the presence or absence of supporting evidence for a claim is insufficient; it is also necessary to accurately assess the logical inference between a claim and its evidence. We propose a new evaluation metric for scientific review comments that assesses the logical inference between claims and evidence. Experimental results show that the proposed method achieves a higher correlation with human scores than conventional methods, indicating its potential to better support the efficiency of the peer-review process.