Linking open-source code commits and MOOC grades to evaluate massive online open peer review

TL;DR

This study links GitHub code commits with MOOC grades to analyze the variability and dynamics of peer review scores in large-scale online courses, revealing that scores are highly variable and not directly tied to code changes.

Contribution

It introduces a novel approach to studying peer review dynamics by linking open-source code data with MOOC grading, providing insights into score variability and re-submission effects.

Findings

Peer review scores are highly variable across submissions.

Scores tend to increase with repeated submissions.

Score changes are not strongly correlated with code modifications.

Abstract

Massive Open Online Courses (MOOCs) have been used by students as a low-cost and low-touch educational credential in a variety of fields. Understanding the grading mechanisms behind these course assignments is important for evaluating MOOC credentials. A common approach to grading free-response assignments is massive scale peer-review, especially used for assignments that are not easy to grade programmatically. It is difficult to assess these approaches since the responses typically require human evaluation. Here we link data from public code repositories on GitHub and course grades for a large massive-online open course to study the dynamics of massive scale peer review. This has important implications for understanding the dynamics of difficult to grade assignments. Since the research was not hypothesis-driven, we described the results in an exploratory framework. We find three distinct clusters of repeated peer-review submissions and use these clusters to study how grades change in response to changes in code submissions. Our exploration also leads to an important observation that massive scale peer-review scores are highly variable, increase, on average, with repeated submissions, and changes in scores are not closely tied to the code changes that form the basis for the re-submissions.