Catch Me if I Can: Detecting Strategic Behaviour in Peer Assessment

TL;DR

This paper develops a statistical test to detect strategic manipulation in peer assessment tasks, such as peer grading and review, and validates its effectiveness through experiments and real-world data.

Contribution

It introduces a principled detection method for strategic behaviour in peer assessment, with strong theoretical guarantees and practical evaluation.

Findings

The test has low false alarm rates.

It effectively detects strategic behaviour in experiments.

The dataset of strategic patterns is publicly released.

Abstract

We consider the issue of strategic behaviour in various peer-assessment tasks, including peer grading of exams or homeworks and peer review in hiring or promotions. When a peer-assessment task is competitive (e.g., when students are graded on a curve), agents may be incentivized to misreport evaluations in order to improve their own final standing. Our focus is on designing methods for detection of such manipulations. Specifically, we consider a setting in which agents evaluate a subset of their peers and output rankings that are later aggregated to form a final ordering. In this paper, we investigate a statistical framework for this problem and design a principled test for detecting strategic behaviour. We prove that our test has strong false alarm guarantees and evaluate its detection ability in practical settings. For this, we design and execute an experiment that elicits strategic behaviour from subjects and release a dataset of patterns of strategic behaviour that may be of independent interest. We then use the collected data to conduct a series of real and semi-synthetic evaluations that demonstrate a strong detection power of our test.