The PeerRank Method for Peer Assessment
Toby Walsh

TL;DR
The paper introduces the PeerRank method for peer assessment, which uses a fixed point approach similar to PageRank to weight grades and incentivize correct grading, showing significant error reduction in synthetic data.
Contribution
It presents a novel PeerRank algorithm that improves peer grading accuracy by incorporating grade-based weighting and fixed point computation.
Findings
Reduces grade prediction error by over 50% in synthetic tests
Provides formal properties like unanimity and no discrimination
Demonstrates promising performance compared to simple averaging
Abstract
We propose the PeerRank method for peer assessment. This constructs a grade for an agent based on the grades proposed by the agents evaluating the agent. Since the grade of an agent is a measure of their ability to grade correctly, the PeerRank method weights grades by the grades of the grading agent. The PeerRank method also provides an incentive for agents to grade correctly. As the grades of an agent depend on the grades of the grading agents, and as these grades themselves depend on the grades of other agents, we define the PeerRank method by a fixed point equation similar to the PageRank method for ranking web-pages. We identify some formal properties of the PeerRank method (for example, it satisfies axioms of unanimity, no dummy, no discrimination and symmetry), discuss some examples, compare with related work and evaluate the performance on some synthetic data. Our results show…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsEducational Technology and Assessment · Machine Learning and Algorithms · Intelligent Tutoring Systems and Adaptive Learning
