Mechanical TA 2: A System for Peer Grading with TA Support

TL;DR

Mechanical TA 2 is an open-source, modular web system designed to facilitate scalable peer grading with TA support, enabling experimentation with grading mechanisms and providing features like calibration, appeals, and feedback.

Contribution

It presents a complete reimplementation of a peer grading system that is scalable, customizable, and suitable for experimental research.

Findings

Supports large-scale peer grading with TA oversight

Enables testing of different peer grading mechanisms

Includes features for calibration, appeals, and feedback

Abstract

Mechanical TA 2 (MTA2) is an open source web-based peer grading application that leverages trusted TA graders to incentivize high-quality peer review. A previous, prototype implementation of MTA proved the value of the concept, but was neither suitable for use at scale nor easily extensible; MTA2 is a complete reimplementation of the system that overcomes these hurdles. MTA2 serves two, interconnected purposes: facilitating practical peer grading and serving as a testbed for experimentation with different peer grading mechanisms. The system is characterized by a modular design that makes customization easy; support for dividing students into different pools based on their peer-grading prowess; mechanisms for automated calibration and spot checking; and the ability for students to appeal grades and to give feedback about individual reviews.