Report-Sensitive Spot-Checking in Peer-Grading Systems

TL;DR

This paper introduces a new peer grading incentive mechanism that reduces TA grading effort by adapting probabilities based on student reports, with proven optimality and applicability to diverse noise models.

Contribution

It presents a novel, adaptive incentive mechanism for peer grading that improves efficiency and is proven to be optimal under various assumptions.

Findings

Mechanism reduces TA grading effort compared to fixed-probability approaches.

Proven necessary and sufficient conditions for mechanism feasibility.

Demonstrated improvements through analytical and empirical analysis.

Abstract

Peer grading systems make large courses more scalable, provide students with faster and more detailed feedback, and help students to learn by thinking critically about the work of others. A key obstacle to the broader adoption of peer grading systems is motivating students to provide accurate grades. The literature has explored many different approaches to incentivizing accurate grading (which we survey in detail), but the strongest incentive guarantees have been offered by mechanisms that compare peer grades to trusted TA grades with a fixed probability. In this work, we show that less TA work is required when these probabilities are allowed to depend on the grades that students report. We prove this result in a model with two possible grades, arbitrary numbers of agents, no requirement that students grade multiple assignments, arbitrary but homogeneous noisy observation of the ground truth which students can pay a fixed cost to observe, and the possibility of misreporting grades before or after observing this signal. We give necessary and sufficient conditions for our new mechanism's feasibility, prove its optimality under these assumptions, and characterize its improvement over the previous state of the art both analytically and empirically. Finally, we relax our homogeneity assumption, allowing each student and TA to observe the ground truth according to a different noise model.