Sem-Detect: Semantic Level Detection of AI Generated Peer-Reviews

TL;DR

Sem-Detect is a novel method that combines textual features and claim-level semantic analysis to accurately distinguish between human-written and AI-generated peer reviews, even when reviews are refined by LLMs.

Contribution

This paper introduces Sem-Detect, a new authorship detection approach that leverages semantic analysis and comparison with AI-generated reviews to identify AI-written peer reviews.

Findings

Sem-Detect achieves a 25.5% improvement over baselines in TPR@0.1% FPR.

It effectively distinguishes human reviews from AI-generated ones, including LLM-refined reviews.

Less than 3.5% of LLM-refined human reviews are misclassified as AI-generated.

Abstract

How can we distinguish whether a peer review was written by a human or generated by an AI model? We argue that, in this setting, authorship should not be attributed solely from the textual features of a review, but also from the ideas, judgments, and claims it expresses. To this end, we propose Sem-Detect, an authorship detection method for peer reviews that operationalizes this principle by combining textual features with claim-level semantic analysis. Sem-Detect compares a target review against multiple AI-generated reviews of the same paper, leveraging the observation that different AI models tend to converge on similar points, while human reviewers introduce more unique and diverse ones. As a result, Sem-Detect is able to distinguish fully AI reviews from authentic human-written ones, including those that have been refined using an LLM but still reflect human judgment. Across a dataset of over 20,000 peer reviews from ICLR and NeurIPS conferences, Sem-Detect improves over the strongest baseline by 25.5% in TPR@0.1% FPR in the binary setting. Moreover, in the three-class scenario, we empirically show that LLM refinement preserves the semantic signals of human reviews, which remain distinct from the patterns exhibited by fully AI-generated text; as a result, fewer than 3.5% of LLM-refined human reviews are misclassified as AI-generated.