Improving Assessment on MOOCs Through Peer Identification and Aligned Incentives

TL;DR

This paper introduces the IPR framework to enhance peer assessment in MOOCs by providing non-blind reviews and incentives, resulting in higher quality feedback and increased peer interaction.

Contribution

The paper proposes a novel peer review framework that improves feedback quality and engagement in MOOC peer assessments.

Findings

IPR yields longer, more useful feedback

IPR increases peer discussion and interaction

IPR outperforms traditional peer assessment methods

Abstract

Massive Open Online Courses (MOOCs) use peer assessment to grade open ended questions at scale, allowing students to provide feedback. Relative to teacher based grading, peer assessment on MOOCs traditionally delivers lower quality feedback and fewer learner interactions. We present the identified peer review (IPR) framework, which provides non-blind peer assessment and incentives driving high quality feedback. We show that, compared to traditional peer assessment methods, IPR leads to significantly longer and more useful feedback as well as more discussion between peers.

Full text

††conference: LAS’17, April 2017, Boston, MA, USA

1 Introduction

Peer assessment in Massive Open Online Courses (MOOCs) affords grading open ended assignments of many students, but this approach often can’t provide the level of feedback that students need. Alternatives such as automated grading enable grading at scale, but require specialized assignment design to facilitate accurate algorithmic judgment, and can’t deal well with open ended task designs, sacrificing student learning for ease of grading [14].

Peer assessment on MOOCs often leads to inaccurate grades and low quality feedback [17], due to laziness, collusion, dishonesty, retaliation and a lack of time, experience or interest [14]. To counter this, humanizing feedback prompt phrasing [12] and offering bonus points to feedback deemed helpful by the receiver [20] have offered improved efficacy. In point systems, if a reviewer provided helpful feedback, the receivers will rate it, however without intrinsic motivation to provide a rating, this leads to single sides systems.

Additionally, MOOC peer reviews are carried out double blind where peers are not made aware of whom they are reviewing or who has reviewed them. This leads to a rising problem in MOOC assessment where due to increased anonymity, there is reduced accountability and eroded community affiliation [16]. Blind peer grading has been practiced in many face to face classroom environments and online learning environments to prevent grading bias and mitigate targeted criticism or bullying. A disadvantage of blind reviews arises in MOOCs when students provide lower quality and less insightful feedback because, being anonymous in review, they are not socially accountable.

Evaluating peers work is a great means of learning. In face to face classroom situations peer evaluation often leads to a conversation where both parties interact richly and gain important understandings about the work, through back and forth communication. MOOCs provide forums for communication and networking, however they are often flooded with community discussion, weeding out paired peer discussions because individual social connections are not readily made.

We introduce the Identified Peer Review (IPR) framework with reduced anonymity and matched incentives to counter these issues of current approaches and increase communication. We evaluated the design with an between subject field experiment. This paper presents preliminary results with a small number of participants (n=87), and describes a future implementation and further evaluation plans.

2 Related Work

Assessment is important to the pedagogy of MOOCs  [4]. MOOCs utilize peer assessment to assess student’s work in a scalable way, not dependent on a one to many teacher to student relationship, but, optimal methods of choosing graders and assignments to grade remains an open question [3]. The benefits of peer assessment include improvement of higher order thinking skills, consolidation of topical knowledge, and individualized feedback for each participant [7]. Giving and getting feedback has been identified as an effective way to learn in online [18] and in the classroom.

However, with the potential benefits, peer grading still faces challenges. Providing accurate grading where the performance of a novice is being judged by other novices is problematic. Students mistrust peer grades and anonymity lead students to be unscrupulous. Reviews are often short and not well considered, which is especially problematic when students work hard on an assignment and receive comments they can’t learn from [21].

Our framework is design based on assessment interactions with visible incentivized peers. Anonymity is commonly practiced in online and in person class peer review systems. Due to the comfort of peers in providing critical feedback [5], the bigger problem in online reviewing is due to increased anonymity and reduced community affiliation [8, 16]. Visible identities lead to more constructive feedback than anonymous ones [23]. Thus through providing non blind review, our framework may encourage accountable reviewing.

Feedback improves performance by changing students’ locus of attention, focusing them on productive aspects of their work [10, 14]. However, not all peers provide great feedback and some leave limited comments with no coherent message for improvement, or they are rogue reviews [17]. Rogue reviews are insufficient reviews caused by laziness, collusion, dishonesty, retaliation, competition, or malevolence [11]. To improve on this, PeerStudio [15] peer assessment system is designed to encourage more feedback comments by showing short tips for writing comments just below the comment box. For example, if a response has no constructive feedback, it may remind students with phrases like: ”Quick check: Is your feedback actionable?” by triggering heuristics word count on feedback [15]. Students see such comments as more useful than rubrics in reviewing [12]. Similar techniques are used to improve the quality of product reviews online [9]. Our framework uses a simple interface design reflecting these lessons by integrating four pointed questions with separate response areas.

In MOOCs, social connections and networking are attractive features for improving learning and providing a platform for desperate learners to interact and learn together. Since MOOCs are often open to the public, the diverse nature of students can be an asset for improving learning performance, increasing innovations, creativity and critical thinking rather than negativity [2, 11]. The cMOOC [19] approach explains peers as a learning source and finds that the more connections achieved in network with diverse perspectives from participants leads to a richer learning environment. By providing feedback and interactions in viewing assignments, projects, and online discussions as opportunities for crowd-sourcing, this approach leads to superior results that otherwise cannot be achieved individually by students (or the instructor) [19]. However, the cMOOC approach requires significant integration and system familiarity, so it is not widespread. Peer assessment is a great way of learning from each others in the community, yet MOOC designs tend to be so complex that no student can simply choose a peer to align with for feedback. Our framework provides a connection and communication mechanism where peers can talk and learn directly. Such discussions and interactions with diverse groups increase the learning significantly [13]. In particular, an HCI course on Coursera encouraged students to post assignments to forums in getting feedback, leading to a conversation of feedback with identified peers and to more connectivity on fellow peers. In this HCI MOOC, more than 75% students were in favor of sharing their assignments on the public [14], where they assert that blind peer review only drives decreased social connection and availability of diverse perspectives.

3 Identified Peer Reviewing Framework

The IPR framework is built upon 4 phases: submit assignments, review peers, rate feedback, receive grades (Figure 1). In the first phase, students submit their assignments. In the second phase, students are given a peer introduction form to provide a short public introduction to be displayed in reviews they conduct. Next students review peer’s work with the review form, an interface with four targeted feedback fields and grade input field. Students then receive feedback on their own work from other peers in the feedback form, which initially shows only the feedback fields, a text input to converse with the reviewer, and a five point scale to rate the usefulness of the feedback. Once all rating is finished, students will receive the grades their reviewers assigned them.

Each student reviews three other students’ work, and receives reviews from three students. Unlike most peer reviewing systems, IPR does not randomize peer review assignment, instead aligning incentives by matching a student with reviewers based on the usefulness of that student’s previous feedback. Students are motivated to provide their peers with high quality feedback to get high quality peer feedback in return. The next round of review allocations are made when feedback has been rated. Only after a student has received and rated all their feedback are grades made visible, so good feedback ratings can’t be bought with inflated grades.

4 Evaluation and Results

We hypothesize that having identifiable peers and matched incentives will increase the feedback quality and lead to more communication than blind peer reviewing. To test this, we conducted a between subject experiment with 3 conditions.

Control condition: Blind peer review, randomized reviewers. 2. 2.

IPR random condition: Identified peer review, randomized reviewers. 3. 3.

IPR incentive condition: Identified peer review, incentivized reviewers.

87 participants collected from an online advertisement placed through authors social media channels and participants were randomly assigned one of three conditions. We created an assignment on the subject ”Creativity and Innovation”, requiring no previous lessons or special subject knowledge. We measured how students perform in each condition, and how useful the feedback was to them.

4.1 How does reviewer blindness influence feedback quality?

To test this, we ran an independent t-test using the control condition and the identified condition. Students in the control condition had an average feedback quality score of $1.21$ out of $29$ ($\sigma=.243$), and students in the identified condition had an average score of $2.43$ out of $29$ ($\sigma=1.02$). An unpaired t-test confirmed that students in the identified condition is significant than other: $t(56)=6.24$, $p=0.000$, $\alpha=0.05$. Which means identified peer reviewing had a large positive effect on the usefulness of feedback compared with double-blind peer-reviewing of assignments.

4.2 How does incentive alignment influence feedback quality?

To test this we ran another independent t-test using the identified condition and the aligned incentives condition. Students in the identified condition had an average feedback quality score of $2.43$ out of $29$ ($\sigma=1.08$), and students in the aligned incentives condition had an average score of $2.12$ out of $25$ ($\sigma=0.71$). The t-test confirmed that students in the identified peer grading condition is significant: $t(48)=1.19$, $p=0.009$, $\alpha=0.05$.

A possible impact of the incentive alignment condition would be to make the gap between the best and worst students larger as the best students help each other, and the worst students are never given high quality feedback. In this initial study, we have not seen evidence of this effect, and hope to explore this in more detail in the future.

4.3 Students motivation to communicate

We analyzed the peer reviews done by all 3 conditions from the communicating message box shown in Figure 1. The control group blind peer reviews contained only 29 messages responding to the feedback they received while the identified condition contained 53 and the aligned incentives condition received 49. The control condition group messages did not contain meaningful communication while treatment groups both had students interested in further communicating together. For example control case messages were limited to one or two words such as ”Thanks, OK” while the treatment groups had messages like ”I like to network too, it is great to be connected to someone new, thanks for the advice - KIT.” The message box also gave students means to ask and respond to questions about the assignment or feedback, for example, a student was able to inform their reviewer that a broken link to the assignment for review had been repaired.

5 Discussion and Future work

We aimed to reintroduce identity in a humanize form where peers introduce themselves to each other. This shared identity showed value, driving conversation and improving feedback quality. Inspired by the design studio concept  [22] and cMOOCs where learners make meaningful connections and learn by giving feedback, we designed the IPR framework and witness early results of students’ response, compared to a control condition. At the same time adding incentive matching in which reviewers and those being reviewed both had the incentive to be honest in their responses. We believe these results introduce incentive compatible interaction design [6] to MOOCs and that this kind of design offers significant opportunities for ongoing improvement in this field.

This research will continue through testing the influence IPR has on larger cohorts’ in a fully develop MOOC. For decades, online peer reviewing has been blind, we aimed to integrate identity in assessment on an MOOC, by carefully designing interactions leading students to discuss and have reason to improve their feedback quality. Initial experimentation showed minimal advantage using incentive alignment over randomized review assignment, but both test conditions performed significantly better in review usefulness and in starting conversations. We believe this is a consequence of the small sample size, so we will explore efficiency with a larger cohort where our ultimate goal is to provide effective feedback to students with meaningful connections, so they can benefit from the diverse crowd in a MOOC.