Learning in teams: peer evaluation for fair assessment of individual contributions

TL;DR

This paper introduces a new peer evaluation mechanism for collaborative learning that accurately assesses individual contributions by integrating instructor credibility and eliminating reliance on self-evaluation.

Contribution

The paper proposes a novel peer evaluation method that combines instructor judgment with student evaluations, ensuring fairness and accuracy without depending on self-assessment.

Findings

Mechanism becomes objective if students are truthful

Integrates instructor credibility to weight evaluations

Achieves fair assessment of individual contributions

Abstract

The "free rider" problem has long plagued pedagogies based on collaborative learning. The most common solution to the free rider problem is peer evaluation. As well other existing methods of peer evaluation include self-evaluation --- and hence are prone to grade inflation or, as we show here, are inaccurate in that they do not fairly reward the most hard working student. Another common concern with existing methods of peer evaluation is that students often do not have the necessary skills to evaluate the work of their peers objectively. In this paper, we introduce a new mechanism for peer evaluation that does not rely on self-evaluation, and yet remains accurate, i.e., if all students are completely truthful in their evaluations, then the output of our mechanism becomes an objective truth. At the same time, our mechanism integrates the instructor's judgment with respect to the credibility of students' evaluations. For example, the instructor gives scores to students for writing credible reviews and, in turn, subsequent students' evaluations are weighted according to these instructor scores.

Equations

$$\begin{gathered} t=(t_1,t_2\dots ,t_n),\text{mbox}where{t}_1\ge 0,t_2\ge 0,\dots ,t_n\ge 0 \\ \text{mbox}and{t}_1+t_2+\cdots +t_n=1 \end{gathered}$$

$$\begin{gathered} s=(s_1,s_2\dots ,s_n),\text{mbox}where{s}_1\ge 0,s_2\ge 0,\dots ,s_n\ge 0 \\ \text{mbox}and{s}_1+s_2+\cdots +s_n=1 \end{gathered}$$

$$a_{ij}\ge 0,\sum _{i=1}^n{a}_{ij}=1,s_i=\frac{1}{n}\sum _{j=1}^n{a}_{ij}.$$

$$a_{ij}\ge 0,a_{ii}=0,\sum _{i=1}^n{a}_{ij}=1,s_i=\frac{1}{n}\sum _{j=1}^n{a}_{ij}.$$

$$t=\left(\frac{1}{2},\frac{1}{4},\frac{1}{4}\right).$$

$$A=\left[\begin{array}{ccc}0& 2/3& 2/3\\ 1/2& 0& 1/3\\ 1/2& 1/3& 0\end{array}\right],s=\left[\begin{array}{c}4/9\\ 5/18\\ 5/18\end{array}\right]\ne \left[\begin{array}{c}1/2\\ 1/4\\ 1/4\end{array}\right]$$

$$t=\left(\frac{1}{2},\frac{1}{2},0\right).$$

$$A=\left[\begin{array}{ccc}0& 1& 1/2\\ 1& 0& 1/2\\ 0& 0& 0\end{array}\right],s=\left[\begin{array}{c}1/2\\ 1/2\\ 0\end{array}\right]=t,$$

$$\frac{a_{ik}}{a_{ik}+a_{jk}}$$

$$\frac{1}{n-2}\sum _{k\ne i,j}\frac{a_{ik}}{a_{ik}+a_{jk}}$$

$$b_{ij}=\frac{{\sum }_{k\ne i,j}\frac{a_{ik}}{a_{ik}+a_{jk}}}{{\sum }_{k\ne i,j}\frac{a_{jk}}{a_{ik}+a_{jk}}}$$

$$A=\left[\begin{array}{ccc}0& 2& 2\\ 3& 0& 3\\ 5& 5& 0\end{array}\right],B=\left[\begin{array}{ccc}1& 2/3& 2/5\\ 3/2& 1& 3/5\\ 5/2& 5/3& 1\end{array}\right]$$

$$b_{ij}=\frac{{\sum }_{k\ne i,j}\frac{w_k\cdot a_{ik}}{a_{ik}+a_{jk}}}{{\sum }_{k\ne i,j}\frac{w_k\cdot a_{jk}}{a_{ik}+a_{jk}}},$$

$$t=(0.1,0.2,0.3,0.4).$$

$$A=\left[\begin{array}{cccc}0& 1& 11& 1\\ 2& 0& 19& 2\\ 3& 1& 0& 3\\ 4& 1& 39& 0\end{array}\right]$$

$$B=\left[\begin{array}{cccc}1& 0.518987& 0.333333& 0.282051\\ 1.92683& 1& 0.666667& 0.497418\\ 3& 1.5& 1& 0.75\\ 3.54545& 2.01038& 1.33333& 1\end{array}\right]$$

$$b_{12}=\frac{\frac{1\cdot a_{13}}{a_{13}+a_{23}}+\frac{3\cdot a_{14}}{a_{14}+a_{24}}}{\frac{1\cdot a_{23}}{a_{13}+a_{23}}+\frac{3\cdot a_{24}}{a_{14}+a_{24}}}=\frac{\frac{11}{11+19}+\frac{3\cdot 1}{1+2}}{\frac{19}{11+19}+\frac{3\cdot 2}{1+2}}=0.518987$$

$$b_{ij}=\{\begin{array}{ll}1,& \text{mbox}ifi=j,\\ \infty ,& \text{mbox}if{t}_i>0,\\ \text{mbox}undefined,& \text{mbox}ifi\ne j\text{mbox}and{t}_i=0\end{array}$$

$$s={\mathrm{Average}}_j\frac{B_{\ast j}}{{\sum }_{i=1}^n{b}_{ij}},$$

$$a_{ij}=\frac{t_i}{1-t_j}$$

$$b_{ij}=\frac{{\sum }_{k\ne i,j}\frac{w_k\cdot \frac{t_i}{1-t_k}}{\frac{t_i}{1-t_k}+\frac{t_j}{1-t_k}}}{{\sum }_{k\ne i,j}\frac{w_k\frac{t_j}{1-t_k}}{\frac{t_i}{1-t_k}+\frac{t_j}{1-t_k}}}=\frac{{\sum }_{k\ne i,j}\frac{w_k{t}_i}{t_i+t_j}}{{\sum }_{k\ne i,j}\frac{w_k{t}_j}{t_i+t_j}}=\frac{t_i{\sum }_{k\ne i,j}\frac{w_k}{t_i+t_j}}{t_j{\sum }_{k\ne i,j}\frac{w_k}{t_i+t_j}}=\frac{t_i}{t_j},$$

$$c_{ij}=\frac{a_{ij}}{{\sum }_{i=1}^n{a}_{ij}}$$

$$\left|\frac{c_{ij}-t_j}{t_j}\right|$$

$$E_j=\frac{1}{n}\sum _{i=1}^n\left|\frac{c_{ij}-t_j}{t_j}\right|$$

$$0.9s_i+0.05w_i+0.05(1-\min (1,E_i)),$$

$$t=\left(\frac{1}{2},\frac{1}{4},\frac{1}{4}\right).$$

$$A=\left[\begin{array}{ccc}0& 2/3& 1/2\\ 1/2& 0& 1/2\\ 1/2& 1/3& 0\end{array}\right],B=\left[\begin{array}{ccc}1& 1& 2\\ 1& 1& 1\\ 1/2& 1& 1\end{array}\right]$$

$$s=\frac{1}{3}\left[\begin{array}{c}2/5\\ 2/5\\ 1/5\end{array}\right]+\frac{1}{3}\left[\begin{array}{c}1/3\\ 1/3\\ 1/3\end{array}\right]+\frac{1}{3}\left[\begin{array}{c}1/2\\ 1/4\\ 1/4\end{array}\right]$$

Full text

In this paper, we introduce a new mechanism for peer evaluation that does not rely on self-evaluation, and yet remains accurate, i.e., if all students are completely truthful in their evaluations, then the output of our mechanism becomes an objective truth. At the same time, our mechanism integrates the instructor’s judgment with respect to the credibility of students’ evaluations. For example, the instructor gives scores to students for writing credible reviews and, in turn, subsequent students’ evaluations are weighted according to these instructor scores.

Keywords:

peer evaluation, collaborative learning, cooperative learning

MSC 2010

Primary: 97D60, Secondary 91A80

1 Introduction

A vast body of literature exists on methods of assessment in tertiary education — see, for example, [10]. In practice, however, written final exams prevail, even though most students will never take an exam in their life after graduation and therefore exam grades are hardly able to capture the true potential of a student to thrive in a complex work environment.

Even though most people never take formal exams after leaving school, working in teams and writing reports are typical job tasks of modern homo sapiens. Team work and report writing are taught at universities, but grading every individual student fairly based on a team’s report is a challenge. For instance, if all the team members get the same grade, then a free-rider problem may occur (see [7], [5], [1]).

The most obvious solution of the free-rider problem is peer evaluation (see [3]). The simplest and yet popular approach to peer evaluation is letting each student grade contribution of each of the team members in absolute terms, i.e., out of 10, a 100, as A, B, C or in a similar way (see, for example, [4], [8], or [9]). According to our experience, peer evaluation in absolute terms results to almost all students giving maximal scores to each other just not to jeopardize their friends’ final grade. Thavikulwat and Chang criticize the whole idea of peer evaluation in [12] and propose to replace it with a different procedure based on students choosing their preferred group size.

Note that while the free rider problem may plague a variety of collaborative learning activities, such as team-based learning (see [11]), the main scenario that we have in mind is a team of students collaborating on a well-defined list of tasks. The group size then should equal, roughly, the number of individual tasks within the project and therefore varying the group size is not an option.

There exist sophisticated peer evaluation systems. The system described in [1] is based on each student allocating a certain number of points between their teammates. Kauffman et al. introduce in [6] a mixed system where students give each other ratings from a list of nine terms such as ”excellent”, ”very good” etc. but these ratings are then converted into a numeric value by dividing everyone rating by the team’s average.

A rigorous mathematical theory of peer evaluation is outlined in [2]. However, the theoretical work [2] is very broad and aimed at mathematicians — experts in game theory. In this paper, we are going to narrow the scope of the theory and to simplify it so it becomes accessible by educators and education researchers.

A system of peer evaluation is a procedure of calculating the ”true” (at least, as it is perceived by team members) contribution of each of the team members into the common task based on mutual evaluations reported by team members. A system of peer evaluation may or may not have certain desired qualities. One such quality is resistance to grade inflation. A system of peer evaluation that includes self-evaluation as part of the process is prone to grade inflation because students have incentive to overestimate their own contribution. The second desired quality is accuracy. A system of peer evaluation is accurate if it outputs true contributions of each of the team members whenever they all report the truth. As we show below, there exist systems of peer evaluation that are accurate but prone to grade inflation and systems that do not rely on self-evaluation but are not accurate. The third quality that may also be desired of a peer evaluation system is integration of the instructor’s judgment into the system. The reason for integrating the instructor’s judgment into the system is that students either may not have the expertise to evaluate their peers or may not have incentive to do it fairly. The instructor will then serve as moderator.

In this paper, we develop a system of peer evaluation that is accurate, does not rely on self-evaluation, and integrates the instructor’s judgment.

2 Mathematical Theory

We assume that $n$ students collaborate on a common goal of completing a set of well-defined tasks and there exist the objective truth — the share of total work that each student has accomplished. If the true contribution/share of the $i$ th student is $t_{i}$, then the objective truth is the vector

[TABLE]

The vector $t$ is known to students but can’t be observed by the course instructor directly and the system of peer evaluation should motivate students to reveal the truth to the instructor. What students report to the instructor is a matrix $A$ of evaluations of each student by each student. Let entries of this matrix be denoted $a_{ij}$ – evaluation of student $i$ by student $j$. We assume that $a_{ij}\geq 0$ (even though systems of peer evaluation with negative scores exist, one can convert such a system to a system with non-negative scores simply by taking the exponential function of each score).

For each $i$, the vector $A_{i*}=(a_{i1},a_{i2},\dots,a_{in})$ is the vector of evaluations received by student $i$ and, for each $j$, the vector $A_{*j}=(a_{1j},a_{2j},\cdots,a_{nj})$ is the vector of evaluations reported by student $j$.

2.1 Mechanism

A mechanism (term adopted from [2]) is an algorithm of calculating the vector of perceived students’ contributions / shares of workload

[TABLE]

from the matrix $A=(a_{ij})_{1\leq i\leq n,1\leq j\leq n}$. Note that the output of the mechanism, i.e., the vector $s$ of perceived contributions may or may not be equal to the vector $t$ of true contributions.

A mechanism may or may not rely on self-evaluation. If students are not required to report a self-evaluation, we let $a_{ii}=0$ for all $i$. A mechanism that relies on self-evaluation will be prone to grade inflation — a student will be tempted to give an unfairly high evaluation to himself.

A mechanism is accurate if outputs true contributions of each team member whenever all students are truthful. In other words, if, for each $j$, the vector $A_{*j}=(a_{1j},a_{2j},\cdots,a_{nj})$ reported by student $j$ is proportional to the vector of true contributions $t=(t_{1},t_{2},\dots,t_{n})$, then we must get $s=t$.

If the course instructor’s purpose is to fairly evaluate each student’s individual contribution to the team effort, the mechanism should be accurate and should not rely on self-evaluation. However, as we show below, mechanisms that are widely used in practice, usually have one of these two qualities, but not both. Currently, mechanisms used in practice are mostly variations of one of the following two.

Pie-to-all mechanism

The pie-to-all mechanism works as follows. Each student gets one pie and then distributes her pie among all team members, including herself, in proportion to their contribution to the team effort. The final perceived contribution of a team member is the average piece of pie that he received from all the team members. It means that we have

[TABLE]

It is easy to show that pie-to-all is an accurate mechanism. However, it relies on self-evaluation and hence is prone to grade-inflation. Clearly, the best strategy to student $j$ is to report $a_{jj}=1$ and $a_{ij}=0$ for $i\neq j$.

Still, the pie-to-all mechanism has been used in practice — see, for example, [6].

Pie-to-others mechanism

The pie-to-others mechanism works as follows. Each student gets one pie and then distributes her pie among all her teammates, not including herself, in proportion to their contribution to the team effort. The final perceived contribution of a team member is the average piece of pie that he received from all the team members. It means that we have

[TABLE]

The only difference with the pie-to-all mechanism (1) is the absence of self-evaluation, which is expressed by $a_{ii}=0$.

The pie-to-others is a very popular mechanism, probably the most popular one. Its clear advantage is that it does not rely on self-evaluation. However, a huge issue with the pie-to-others mechanism is that it is not accurate as the following example shows.

Example 1

Suppose that we have a team of three students and the vector of true contributions is

[TABLE]

The matrix of peer-evaluations and the the vector of perceived contributions are

[TABLE]

□

In our experience, real students, at least those whose major is mathematics, understand very well that the pie-to-others mechanism is not accurate. Students will not be happy if a large portion of their grade comes from this mechanism.

However, the inaccuracy of pie-to-others is not the only problem of this mechanism. Well, pie-to-others still gives the highest score to the most hard-working student in a team, even though that score may not accurately reflect the true contribution. Let’s demonstrate the other issue by an example.

Example 2

Consider a hypothetical team of three students whose vector of true contributions is

[TABLE]

The matrix of peer evaluations and the the vector of perceived contributions are

[TABLE]

which means that now all the students are fairly rewarded. □

Thus it is profitable for strongest students to just do all the work by themselves without letting their teammates do anything. It’s then (and only then) that they will be fairly rewarded for their hard work. This behavior is very common in practice and a serious weakness of the pie-to-others mechanism is that it incentivizes such behavior.

3 Results

In this section, we outline a new mechanism that is accurate but does not rely on self-evaluation.

3.1 Auxiliary matrix

Let $a_{ij}$ be raw evaluations of student $i$ by student $j$ for $i,j\in\{1,2,\dots,n\}$. We are going to construct an auxiliary matrix $B$ whose entries $b_{ij}$ show relative contributions of students $i$ and $j$, the ratio of $i$’s contribution to $j$’s contribution according to other team members. If $k$ is any student other than $i$ and $j$, then

[TABLE]

is the share of $i$’s contribution in the combined $i$’s and $j$’s contribution according to $k$. Therefore,

[TABLE]

is the average share of $i$’s contribution in the combined $i$’s and $j$’s contribution according to their teammates. Thus,

[TABLE]

is the average ratio of $i$’s contribution to $j$’s contribution according to their teammates.

Note that (3) does not change if we multiply all evaluations reported by a student by a constant. It means that it is not necessary to normalize the matrix of peer-evaluations $A$ to compute the auxiliary matrix $B$.

Example 3

Consider a team of 3 students with true contributions $t=(0.2,0.3,0.5)$. The matrix of peer-evaluations and the auxiliary matrix are

[TABLE]

Note that each column of the matrix $B$ is proportional to $t$. □

3.2 Instructor’s judgment

It is not realistic to expect that actual students would be completely truthful and absolutely precise in their evaluations. In our experience, students who contribute least of all usually tend not to put too much thought into evaluations that they submit and often simply give equal scores to everyone. It is therefore important to have some procedure of discrediting evaluations that are not trustworthy.

In our mechanism, students not only give numeric evaluations to each other, but also provide justifications, i.e., write short reports on each of their teammates. The instructor will then read these reports and give students grades for writing the good trust-worthy reports. Let $w_{i}$ be the grade given to student $i$ by the instructor for writing reports.

We will now modify the construction of the auxiliary matrix $B$ in order to integrate trustworthiness of students’ reports into it. Specifically, (4) is replaced with

[TABLE]

i.e., the average is replaced with weighted average.

Example 4

Consider four students and true contributions

[TABLE]

Suppose also that the vector of grades for writing reports is $w=(4,0,1,3)$. Probably, student 1 wrote nice trustworthy reports, student 2 did not write anything, student 3 wrote something short and not very meaningful, and student 4 wrote meaningful reports but missed some points. At the same time, students 1 and 4 are truthful in their evaluations, student 2 gave same scores to everyone and student 3 was not completely truthful so that the matrix of raw evaluations is

[TABLE]

Then the auxiliary matrix is

[TABLE]

For instance,

[TABLE]

□

3.3 Missing values

In practice, some students fail to submit evaluations of their teammates, i.e., a real matrix $A$ may have missing values. In this case, the instructor will give a zero score for reports to students who did not write any reports and impute their missing numeric evaluations with equal scores. Such non-existing evaluations will be then automatically discarded.

3.4 Main mechanism

To calculate ”true” individual contributions of each team member from a matrix $A$ of peer evaluations, we let the instructor grade reports written by each student on their teammates. Let $w_{k}$ be the grade given to student $k$ by the instructor, we calculate the auxiliary matrix $B$ by (5). Then the entry $b_{ij}$ of the auxiliary matrix is the relative contribution of student $i$ to student $j$ according to their teammates. We also have $b_{ii}=1$ for all $i$ and $b_{ij}=\frac{1}{b_{ji}}$.

Note that if everyone were completely truthful, then each column of the matrix $B$ would be proportional to the true contributions of the team members. In practice, however, columns of the matrix $B$ may not be proportional to each other due to difference in students’ opinions. Another practical issue is that there may be students who don’t contribute at all. If $j$ is such a student, i.e., $t_{j}=0$, then we would have

[TABLE]

The final scores are computed from the matrix $B$ by

[TABLE]

where the average is taken over all columns $j$ that do not have infinite or missing values. Note that we normalize each column $B_{*j}$ dividing it by the sum of its entries, to ensure that sum of entries of the vector $s$ is $1$.

Theorem 1

Suppose that at least two students made positive contribution to the team’s work and that there are at least three students who evaluated each of their teammates and got positive grades from the instructor for reports that they wrote. Then the mechanism given by (6) is accurate. □

Proof

The assumptions are needed to ensure that the matrix $B$ is well-defined and has at least two columns with finite entries. The rest is straightforward — to show that the mechanism is accurate, we need to prove that its output is the objective truth assuming that all evaluations are truthful. If all the students report the objective truth, then we have

[TABLE]

From (5), we get

[TABLE]

which completes the proof. ■

3.5 Evaluating truthfulness

Although our mechanism is accurate and does not rely on self-evaluation, there is still a practical consideration that it does not address. Sometimes, two friends may cheat the system by giving unfairly high evaluations to each other. If they write convincing reports, the course instructor will overlook the deceit.

The following method was proposed in [2] to discourage such behavior. We actually require students to report self-evaluations, i.e., we have $a_{ii}>0$. Even though self-evaluations are discarded when the mechanism output is computed according to (6), all evaluations reported by a student, including the self-evaluation, are used to measure to which extend evaluations $A_{*j}$ reported by student $j$ deviate from the final scores $s$.

Let us introduce the notation

[TABLE]

for contribution of student $i$ according to student $j$. Then

[TABLE]

is the relative error of evaluation of student $i$ by student $j$ and

[TABLE]

is the average relative error of student $j$’s evaluations.

3.6 Summary of our mechanism

In practice, the main score computed by the mechanism according to (6) constitute 90% of the final score, with 5% given for writing reports and 5% given for consistency of evaluations submitted by a student with scores calculated by the main mechanism. Thus the final score given to student $i$ is

[TABLE]

where $s_{i}$ is the output of the main mechanism computed according to (6), $w_{i}$ is the score given by the instructor to student $i$ for reports that $i$ wrote, and $E_{i}$ is the average relative error of $i$ th evaluations found according to (7).

Note that the average of scores given by (8) is usually smaller than $1$ and equals $1$ if and only if all evaluations submitted by students are completely truthful. Another remark is that the weights $0.9$, $0.05$, and $0.05$ for the output of the main mechanism, reports, and consistency of evaluations with output of the main mechanism are not justified by data or theory and finding ”right” weights may be a subject of future research.

4 Discussion

Our mechanism is accurate. However, even though it does not rely on self-evaluation, it may be possible to manipulate it.

Example 5

Suppose that we have a team of three students and the vector of true contributions is

[TABLE]

Suppose that the third student decided to manipulate the mechanism and reported incorrect evaluations by claiming that the first and the second student contributed equally. Assume also that the course instructor did not see it and graded all reports as being equally trust-worthy. The matrix of peer-evaluations and the auxiliary matrix are

[TABLE]

The vector of percieved contributions is then

[TABLE]

Then the perceived contribution of the third student is $\frac{47}{180}>\frac{1}{4}$. It means that by changing his evaluations of other students, he changed his own score. □

Note that although manipulating our mechanism is not as straightforward as manipulating the pie-to-all mechanism, it is still possible.

A mechanism is called incentive compatible if lying does not improve one’s own score given that others tell the truth. In other words, a mechanism is incentive compatible if the collective truth-telling is a Nash equilibrium. A mechanism is not manipulable if changing evaluations of other students one reports does not change one’s own score. Note that a mechanism that is not manipulable is automatically incentive compatible, but not vice versa.

Our mechanism is manipulable, but we believe that it will become incentive compatible if a small portion of the final score is given for consistency of scores reported by a student with the team’s perceived contributions. However, determining that small portion (in practice, we use $10\%$, but we don’t have justification for that number) is still an open problem. We don’t know if there exists an accurate mechanism that is not manipulable. We believe that it does not, but proving it is another open problem.