Better Peer Grading through Bayesian Inference

TL;DR

This paper advances peer grading accuracy by incorporating strategic student behavior, handling censored data, and improving interpretability through Bayesian inference and mixed integer programming, validated on synthetic and real data.

Contribution

It extends probabilistic peer grading models to account for strategic behavior, censored data, and interpretability, making Bayesian inference practical for large-scale applications.

Findings

Accurately estimates true grades and student error variability.

Identifies students likely submitting uninformative grades.

Demonstrates robustness of the model across datasets.

Abstract

Peer grading systems aggregate noisy reports from multiple students to approximate a true grade as closely as possible. Most current systems either take the mean or median of reported grades; others aim to estimate students' grading accuracy under a probabilistic model. This paper extends the state of the art in the latter approach in three key ways: (1) recognizing that students can behave strategically (e.g., reporting grades close to the class average without doing the work); (2) appropriately handling censored data that arises from discrete-valued grading rubrics; and (3) using mixed integer programming to improve the interpretability of the grades assigned to students. We show how to make Bayesian inference practical in this model and evaluate our approach on both synthetic and real-world data obtained by using our implemented system in four large classes. These extensive experiments show that grade aggregation using our model accurately estimates true grades, students' likelihood of submitting uninformative grades, and the variation in their inherent grading error; we also characterize our models' robustness.