Removing Bias and Incentivizing Precision in Peer-grading

TL;DR

This paper introduces PEQA, a peer-grading mechanism that removes bias, incentivizes accurate grading, and aligns grader utility with reliability, outperforming existing methods in classroom experiments.

Contribution

PEQA is a novel peer-grading mechanism that ensures bias neutrality, incentivizes truthful and reliable grading, and achieves socially optimal effort levels under private costs.

Findings

PEQA outperforms the median mechanism in classroom tests.

Grader utility increases with grading reliability under PEQA.

PEQA effectively removes bias and incentivizes precision.

Abstract

We study peer-grading with competitive graders who enjoy a higher utility when their peers get lower scores. We propose a new mechanism, PEQA, that incentivizes such graders through a score-assignment rule which aggregates the final score from multiple peer-evaluations, and a grading performance score that rewards performance in the peer-grading exercise. PEQA makes grader-bias irrelevant. Additionally, under PEQA, a peer-grader's utility increases monotonically with the reliability of her grading, irrespective of her competitiveness and how her co-graders act. In a reasonably general class of score assignment rules, PEQA uniquely satisfies this utility- reliability monotonicity. When grading is costly and costs are private information, a modified version of PEQA implements the socially optimal effort choices in an equilibrium of the peer-evaluation game. Data from our classroom experiments confirm our theoretical assumptions and show that PEQA outperforms the popular median mechanism.