Detecting Problem Statements in Peer Assessments

TL;DR

This paper presents machine learning approaches, including neural networks and traditional models, to automatically identify problem statements in peer review comments, achieving high accuracy in detection.

Contribution

It introduces a dataset of over 18,000 labeled review comments and evaluates multiple models, highlighting the effectiveness of neural network architectures like Hierarchical Attention Networks.

Findings

Hierarchical Attention Network achieved 93.1% accuracy

Neural network models outperform traditional models

Support vector machine achieved 89.71% accuracy

Abstract

Effective peer assessment requires students to be attentive to the deficiencies in the work they rate. Thus, their reviews should identify problems. But what ways are there to check that they do? We attempt to automate the process of deciding whether a review comment detects a problem. We use over 18,000 review comments that were labeled by the reviewees as either detecting or not detecting a problem with the work. We deploy several traditional machine-learning models, as well as neural-network models using GloVe and BERT embeddings. We find that the best performer is the Hierarchical Attention Network classifier, followed by the Bidirectional Gated Recurrent Units (GRU) Attention and Capsule model with scores of 93.1% and 90.5% respectively. The best non-neural network model was the support vector machine with a score of 89.71%. This is followed by the Stochastic Gradient Descent model and the Logistic Regression model with 89.70% and 88.98%.