ReviewRobot: Explainable Paper Review Generation based on Knowledge Synthesis

TL;DR

ReviewRobot is an AI system that automatically generates detailed, evidence-based, and explainable reviews for academic papers by leveraging knowledge graphs and template-based natural language generation.

Contribution

It introduces a novel method combining knowledge graph comparison and template-based comment generation for automated, explainable paper review synthesis.

Findings

ReviewRobot achieves 71.4%-100% review score prediction accuracy.

41.7%-70.5% of generated comments are valid and constructive.

Generated comments outperform human reviews in 20% of cases.

Abstract

To assist human review process, we build a novel ReviewRobot to automatically assign a review score and write comments for multiple categories such as novelty and meaningful comparison. A good review needs to be knowledgeable, namely that the comments should be constructive and informative to help improve the paper; and explainable by providing detailed evidence. ReviewRobot achieves these goals via three steps: (1) We perform domain-specific Information Extraction to construct a knowledge graph (KG) from the target paper under review, a related work KG from the papers cited by the target paper, and a background KG from a large collection of previous papers in the domain. (2) By comparing these three KGs, we predict a review score and detailed structured knowledge as evidence for each review category. (3) We carefully select and generalize human review sentences into templates, and apply these templates to transform the review scores and evidence into natural language comments. Experimental results show that our review score predictor reaches 71.4%-100% accuracy. Human assessment by domain experts shows that 41.7%-70.5% of the comments generated by ReviewRobot are valid and constructive, and better than human-written ones for 20% of the time. Thus, ReviewRobot can serve as an assistant for paper reviewers, program chairs and authors.