Improving essay peer grading accuracy in MOOCs using personalized weights from student's engagement and performance

TL;DR

This study explores how personalized weights based on student engagement and performance can improve the accuracy of peer grading in MOOCs, especially when using median aggregation, demonstrated through analysis of a philosophy course.

Contribution

It introduces a method to compute personalized weights from engagement and performance data to enhance peer grading accuracy without complex calibration.

Findings

Weighted median aggregation improves score validity.

Performance-based weights increase correlation with instructor grades.

Analysis conducted on data from a philosophy MOOC.

Abstract

Most MOOC platforms either use simple schemes for aggregating peer grades, e.g., taking the mean or the median, or apply methodologies that increase students' workload considerably, such as calibrated peer review. To reduce the error between the instructor and students' aggregated scores in the simple schemes, without requiring demanding grading calibration phases, some proposals compute specific weights to compute a weighted aggregation of the peer grades. In this work, and in contrast to most previous studies, we analyse the use of students' engagement and performance measures to compute personalized weights and study the validity of the aggregated scores produced by these common functions, mean and median, together with two other from the information retrieval field, namely the geometric and harmonic means. To test this procedure we have analysed data from a MOOC about Philosophy. The course had 1059 students registered, and 91 participated in a peer review process that consisted in writing an essay and rating three of their peers using a rubric. We calculated and compared the aggregation scores obtained using weighted and non-weighted versions. Our results show that the validity of the aggregated scores and their correlation with the instructors grades can be improved in relation to peer grading, when using the median and weights are computed according to students' performance in chapter tests.