Optimizing Peer Grading: A Systematic Literature Review of Reviewer Assignment Strategies and Quantity of Reviewers

TL;DR

This systematic review analyzes reviewer assignment strategies and review quantity in peer assessment, highlighting their impact on accuracy, fairness, and educational value, and identifying optimal practices based on empirical evidence.

Contribution

The paper provides a comprehensive synthesis of 87 studies on reviewer assignment and review quantity, identifying common strategies, trade-offs, and recommendations for effective peer assessment.

Findings

Random assignment leads to inconsistent grading.

Competency-based strategies improve fairness.

Three to five reviews per submission balance accuracy and workload.

Abstract

Peer assessment has established itself as a critical pedagogical tool in academic settings, offering students timely, high-quality feedback to enhance learning outcomes. However, the efficacy of this approach depends on two factors: (1) the strategic allocation of reviewers and (2) the number of reviews per artifact. This paper presents a systematic literature review of 87 studies (2010--2024) to investigate how reviewer-assignment strategies and the number of reviews per submission impact the accuracy, fairness, and educational value of peer assessment. We identified four common reviewer-assignment strategies: random assignment, competency-based assignment, social-network-based assignment, and bidding. Drawing from both quantitative data and qualitative insights, we explored the trade-offs involved in each approach. Random assignment, while widely used, often results in inconsistent grading and fairness concerns. Competency-based strategies can address these issues. Meanwhile, social and bidding-based methods have the potential to improve fairness and timeliness -- existing empirical evidence is limited. In terms of review count, assigning three reviews per submission emerges as the most common practice. A range of three to five reviews per student or per submission is frequently cited as a recommended spot that balances grading accuracy, student workload, learning outcomes, and engagement.