Argument Rarity-based Originality Assessment for AI-Assisted Writing

TL;DR

This paper introduces AROA, a framework for automatically assessing argumentative originality in essays by measuring rarity across multiple dimensions, revealing a trade-off between quality and originality and highlighting differences between human and AI-generated texts.

Contribution

The paper presents a novel framework, AROA, that quantifies argumentative originality through multiple rarity-based components, advancing automated essay evaluation methods.

Findings

AI essays have high quality scores but low claim rarity compared to humans.

There is a negative correlation between text quality and claim rarity.

The four originality components are largely independent, capturing different aspects of originality.

Abstract

This study proposes Argument Rarity-based Originality Assessment (AROA), a framework for automatically evaluating argumentative originality in student essays. AROA defines originality as rarity within a reference corpus and evaluates it through four complementary components: structural rarity, claim rarity, evidence rarity, and cognitive depth, quantified via density estimation and integrated with quality adjustment. Experiments using 1,375 human essays and 1,000 AI-generated essays on two argumentative topics revealed three key findings. First, a strong negative correlation (r = -0.67) between text quality and claim rarity demonstrates a quality-originality trade-off. Second, while AI essays achieved near-perfect quality scores (Q = 0.998), their claim rarity was approximately one-fifth of human levels (AI: 0.037, human: 0.170), indicating that LLMs can reproduce argumentative structure but not semantic originality. Third, the four components showed low mutual correlations (r = 0.06--0.13 between structural and semantic dimensions), confirming that they capture genuinely independent aspects of originality. These results suggest that writing assessment in the AI era must shift from quality to originality.