Polarity in the Classroom: A Case Study Leveraging Peer Sentiment Toward Scalable Assessment

TL;DR

This paper explores using sentiment analysis and aspect extraction on peer review comments to improve grading reliability and information quality in large online courses.

Contribution

It introduces a domain-specific sentiment analysis and aspect-informed review form to leverage student comments for better assessment validation.

Findings

Sentiment analysis correlates with grading accuracy.

Aspect extraction helps tailor review forms to student feedback.

Analysis of 6800 reviews demonstrates potential for sentiment in grading.

Abstract

Accurately grading open-ended assignments in large or massive open online courses (MOOCs) is non-trivial. Peer review is a promising solution but can be unreliable due to few reviewers and an unevaluated review form. To date, no work has 1) leveraged sentiment analysis in the peer-review process to inform or validate grades or 2) utilized aspect extraction to craft a review form from what students actually communicated. Our work utilizes, rather than discards, student data from review form comments to deliver better information to the instructor. In this work, we detail the process by which we create our domain-dependent lexicon and aspect-informed review form as well as our entire sentiment analysis algorithm which provides a fine-grained sentiment score from text alone. We end by analyzing validity and discussing conclusions from our corpus of over 6800 peer reviews from nine courses to understand the viability of sentiment in the classroom for increasing the information from and reliability of grading open-ended assignments in large courses.