Co-Attention Based Neural Network for Source-Dependent Essay Scoring

TL;DR

This paper introduces a co-attention neural network for source-dependent essay scoring, improving accuracy and aligning well with expert judgments across multiple datasets.

Contribution

It proposes a novel co-attention mechanism for better importance learning in source-dependent essay scoring models.

Findings

Outperforms baseline models on two datasets

Provides reliable score predictions

Attention aligns with expert opinions

Abstract

This paper presents an investigation of using a co-attention based neural network for source-dependent essay scoring. We use a co-attention mechanism to help the model learn the importance of each part of the essay more accurately. Also, this paper shows that the co-attention based neural network model provides reliable score prediction of source-dependent responses. We evaluate our model on two source-dependent response corpora. Results show that our model outperforms the baseline on both corpora. We also show that the attention of the model is similar to the expert opinions with examples.

Tables

Table 1. Table 1: The holistic score distribution of ASAP.

Prompt	3	4	5	6
Score 0	39	311	24	44
	(2%)	(18%)	(1%)	(3%)
Score 1	607	636	302	167
	(35%)	(36%)	(17%)	(9%)
Score 2	657	570	649	405
	(38%)	(32%)	(36%)	(23%)
Score 3	423	253	572	817
	(25%)	(14%)	(32%)	(45%)
Score 4	NA	NA	258	367
			(14%)	(20%)
Total	1726	1770	1805	1800

Table 2. Table 2: The Evidence score distribution of RTA.

Prompt	$RT{A}_{MVP}$	$RT{A}_{Space}$
Score 1	852	538
	(29%)	(26%)
Score 2	1197	789
	(40%)	(38%)
Score 3	616	512
	(21%)	(25%)
Score 4	305	237
	(10%)	(11%)
Total	2970	2076

Table 3. Table 3: Hyper-parameters of training.

Layer	Parameter Name	Value
Embedding	Embedding dimension	50
Word-CNN	Kernel size	5
	Number of filters	100
Sent-LSTM	Hidden units	100
Modeling	Hidden units	100
Dropout	Dropout rate	0.5
Others	Epochs	100
	Batch size	100
	Initial learning rate	0.001
	Momentum	0.9

Table 4. Table 4: The performance (QWK) of the baselines and our model. ∗ ∗ \ast indicates that the model QWK is significantly better than the SELF-ATTN ( p < 0.05 𝑝 0.05 p<0.05 ). † † \dagger indicates that the model QWK is significantly better than the SG ( p < 0.05 𝑝 0.05 p<0.05 ). The best results in each row are in bold.

Prompts	SVR	BLRR	SG	SELF-ATTN	CO-ATTN
$RT{A}_{MVP}$	NA	NA	0.653	0.681$†$	0.697$\ast †$
$RT{A}_{Space}$	NA	NA	0.632	0.669$†$	0.684$\ast †$
$ASA{P}_3$	0.630	0.621	NA	0.677	0.697$\ast$
$ASA{P}_4$	0.749	0.784	NA	0.807	0.809
$ASA{P}_5$	0.782	0.784	NA	0.806	0.815
$ASA{P}_6$	0.771	0.775	NA	0.809	0.812

Table 5. Table 5: Example attention scores of essay sentences.

No.	Sentences	Attention
1	Life in Kenya is hard.	0.00173
2	In this essay I will give my top 3 reasons why.	0.00174
3	Because like I said, we have more advanced & better & more qualified materials than them, and these days kids & adults are spoiled, we have phones stores, houses & even shoes and clothes.	0.00243
4	The author did convince me that winning the fight against poverty is achievable in our lifetime because she showed me how many people in Sauri, Kenya need our help against poverty.	0.00229
5	Water is connected to the hospitals.	0.02936
6	So the farmer don’t have to worry all the time that him or his family won’t have enough food to eat and the farmer have to worry that their kids will get hungry and then sick.	0.05580
7	The hospital aslo has water and electricity.	0.07746
8	Also, there were no school fees, and the school now serves lunch for the students because they didn’t have any midday meals to provide them with energy they need to help them with the rest of their days.	0.19483
9	In 2008 though, when they checked for progress, the hospital had medicine, free of charge, with running water and electricty.	0.20177
10	Also farmers could not afford fertilizer and irrigation but now they placed irrigation and have them fertilizer for the crops.	0.25855

Equations

$$p_i=g([w_i:w_{i+k-1}]\cdot U_p+b_p)$$

$$m_i=tanh(U_m\cdot p_i+b_m)$$

$$v_i=\frac{e^{u_v\cdot m_i}}{\sum e^{u_v\cdot m_j}}$$

$$s=\sum v_i{p}_i$$

$$f_t=\sigma (W_f\cdot [h_{t-1},s_t]+b_f)$$

$$i_t=\sigma (W_i\cdot [h_{t-1},s_t]+b_i)$$

$${\tilde{c}}_t=tanh(W_c\cdot [h_{t-1},s_t]+b_c)$$

$$c_t=f_t\ast c_{t-1}+i_t\ast {\tilde{c}}_t$$

$$o_t=\sigma (W_o\cdot [h_{t-1},s_t]+b_o)$$

$$h_t=o_t\ast tanh(c_t)$$

$$Sim=W_{sim}^T\cdot [h{e}_t;h{a}_j;h{a}_t\ast h{a}_j^T]+b_{sim}$$

$$a_{ea}=softmax(Sim)$$

$${\tilde{H}}_a=a_{ea}{H}_a$$

$$a_{ae}=softmax(ma{x}_{col}(Sim))$$

$${\tilde{h}}_e=a_{ae}^T{H}_e$$

$$G=[H_e;{\tilde{H}}_a;H_e\ast {\tilde{H}}_a;H_e\ast {\tilde{H}}_e]$$

$$y=sigmoid(W_{o}M+b_o)$$

$$mse(y,y^{\prime })=\frac{1}{N}\sum _{i=1}^N(y_i-y_i^{\prime }{)}^2$$

Full text

1 Introduction

Manually grading students’ essays is labor intensive. Therefore, many automated essay scoring (AES) methods have been developed to support grading essays at scale. However, in different grading tasks, the information required by an AES system is different. For example, if a system needs to assign a holistic score to the essay, the system needs to take all information into account. In contrast, if a system needs to assign a score for one specific aspect of the essay (e.g. use of evidence), the system needs to ignore some information. Also, if an essay is a source-dependent essay, the system needs to exploit knowledge of the source article.

This paper focuses on source-dependent essay assessment. In this task, students read a source article before writing the essay, and assessment involves recognizing and analyzing references to the article in the essay. We propose a new type of co-attention based neural network model tailored to source-dependent grading, then use two source-dependent essay corpora to evaluate our model. Our first corpus contains the four source-dependent essay sets in the Automated Student Assessment Prize (ASAP) corpus111https://www.kaggle.com/c/asap-aes. The ASAP grading task is to assign a holistic score to each essay. The second corpus uses the Response to Text Assessment (RTA) Correnti et al. (2013) to assess students’ analytic writing skills. Instead of evaluating holistic writing skills, the RTA was designed to evaluate students’ writing skills along five dimensions: Analysis, Evidence, Organization, Style, and MUGS (Mechanics, Usage, Grammar, and Spelling). Our grading task for this corpus is to assign an Evidence score to each essay, by evaluating students’ ability to find and use evidence from a source article to support their claims.

The main contributions of this paper are as follows. First, we introduce a co-attention based neural network model that is fully automated and does not need any expert effort to encode knowledge of a source article. Second, our co-attention based neural network model extends prior work by designing the model to take a source article into account during grading. Third, we apply our model to the subset of source-dependent responses tasks in the ASAP corpus and show that the model outperforms a previous neural network model developed for the full corpus. Fourth, we show that our model also performs well on the RTA task and again significantly outperforms our baseline neural net model. Last, we use examples to show that our model can assign reasonable attention scores to different sentences in the essay.

In the following sections, we first present related research. Then we describe our tasks by introducing the ASAP corpus and the RTA corpus. Next, we explain the structure of our co-attention based neural network model. Finally, we discuss the results of our experiments and future plans.

2 Related Work

Previous research in AES needed feature engineering. In very early work, Page (1968) developed an AES tool named Project Essay Grade (PEG) by only using linguistic surface features. A more recent well-known AES system is E-Rater Burstein et al. (1998), which employs many more natural language processing (NLP) technologies. Later, Attali and Burstein (2004) released E-Rater V2, where they created a new set of features to represent linguistic characteristic related to organization and development, lexical complexity, prompt-specific vocabulary usage, etc. Similarly to Page (1968), this system used regression equations for assessment of student essays. One limitation of all of the above models is that all need handcrafted features for training the model. In contrast, our model uses a neural network for the AES task and thus does not require feature engineering.

Recently, neural network models have been introduced into AES, making the development of handcrafted features unnecessary or at least optional. Alikaniotis et al. (2016) and Taghipour and Ng (2016) presented AES models that used Long Short Term Memory (LSTM) networks. Differently, Dong and Zhang (2016) used a Convolutional Neural Network (CNN) model for essay scoring by applying two CNN layers on both the word level and then sentence level. Later, Dong et al. (2017) presented another work that uses attention pooling to replace the mean over time pooling after the convolutional layer in both word level and sentence levels. However, none of these neural network grading models consider the source article if it exists. In this paper, we introduce a neural network model that takes the source article into account by using a co-attention mechanism instead of the self-attention mechanism of prior work.

Our work not only focuses on essay assessment using a holistic score, but also evaluates a particular dimension of argument-oriented writing skills, namely use of Evidence. Louis and Higgins (2010) analyze only the content of essays by detecting off-topic essays. Ong et al. (2014) used argumentation mining techniques to evaluate if students use enough evidence to support their positions. However, these two prior studies are not suitable for our task because they did not measure the use of content or evidence from a source article. With respect to source-based dimensional essay analysis, Rahimi et al. (2014, 2017) developed a set of rubric-based features that compared a student’s essay and a source article in terms of number of related words or paraphrases. Zhang and Litman (2017) improved their model by introducing word embedding into the feature extraction process to extract relationships previously missed due to lexical errors or use of different vocabulary. However, in both of these studies, human effort was still necessary for pre-processing the source article, for example, by having experts manually create a list of important words and phrases in the article which the system would compare with features extracted from the student’s essay. In contrast, our work does not need any human effort to analyze the source article before essay grading. Although Rahimi and Litman (2016) investigated extracting example lists by using LDA (Blei et al., 2003) model, the data-driven model missed an example when there was no essay mentioning the example. Klebanov et al. (2014) predicted which parts of the source material were important and that students needed to use in their essays. The essay score is required to obtain the content importance for their work, but our work does not need to know the essay score while identifying the content importance.

3 Data

We use two different essay corpora in our experiments: source-based essays from the ASAP corpus, and source-based RTA essays. While the full ASAP corpus contains essays in response to 8 different prompts, we use only essays in response to the 4 source-dependent prompts. The gold standard ASAP assessment is a holistic score. In contrast, the gold standard assessment in the RTA corpus is an Evidence score. In particular, the assessment only considers how students use evidence from a source article to support their claims; the assessement thus ignores the lexical and syntactic mistakes made by students and the organization of the essay when assessing the evidence dimension.

3.1 ASAP

The Automated Student Assessment Prize (ASAP) corpus consists of written responses to 8 prompts. Among them, prompts 3, 4, 5, and 6 are source-dependent which means students read an article before writing their essays. Since the scores assigned to essays are holistic, assessment considers the overall quality of the essay, not just a specific dimension. Figure 1 contains an excerpt from an ASAP source article and the associated Prompt 5.

In this paper, we only focus on prompts 3, 4, 5, and 6 (denoted by $ASAP_{3}$, $ASAP_{4}$, $ASAP_{5}$, and $ASAP_{6}$ respectively), because they are source-dependent responses. In ASAP, different prompts have different score ranges. The score range of $ASAP_{3}$ and $ASAP_{4}$ is 0 to 3, while the range of $ASAP_{5}$ and $ASAP_{6}$ is 0 to 4. Figure 2 shows an excerpt of an essay with score of 4 for $ASAP_{5}$. The score distribution is shown in Table 1.

3.2 RTA

The RTA corpora were collected from upper elementary level students, as described by Correnti et al. (2013). There are two forms of RTA based on different articles that students read before writing essays. The first article is from Time for Kids about the Millennium Villages Project, an effort by the United Nations to end poverty in a rural village in Sauri, Kenya; we refer to it as $RTA_{MVP}$. The other article talks about the importance of space exploration; we refer to refer it as $RTA_{Space}$. Figure 3 shows an excerpt from the $RTA_{MVP}$ article and the associated essay writing prompt. Bolded text spans in the article excerpt are pieces of evidence that our experts (School of Education RTA team members) manually labeled as being important for students to include in their essays.

Evidence usage in each RTA essay was scored on a scale of 1 to 4 (low to high). The distribution of Evidence scores is shown in Table 2. Figure 4 shows a student essay with a score of 3. Our experts manually bolded all pieces of evidence found in this essay.

4 Model

Our network is inspired by the hierarchical neural network model presented by Dong et al. (2017). In their model, they considered each essay as a sequence of sentences rather than a sequence of words. Their model has three parts. First, they used a convolutional layer and attention pooling layer to get sentence representation. Second, they used an LSTM layer and another attention pooling layer for document representation. Finally, they used a sigmoid layer for score prediction.

Differently from their model, our model replaces the attention pooling layer for document representation with a bi-directional attention flow layer and an additional modeling layer (Seo et al., 2017). By doing so, our model considers students’ essays associated with a source article and this attention mechanism captures the relationship between the essay and the source article. In particular, a higher attention score will be assigned to sentences that are mentioned in the article but less mentioned in other essays. Our model is a hierarchical neural network and consists of seven layers. Figure 5 shows the structure of our network. The layers in the dashed box were presented by Dong et al. (2017). The sentence level co-attention layer was presented by Seo et al. (2017).

4.1 Word Embedding Layer

This layer maps each word in sentences to a high dimension vector. We use the GloVe pre-trained word embeddings (Pennington et al., 2014) to obtain the word embedding vector for each word. It was trained on 6 billion words from Wikipedia 2014 and Gigaword 5. It has 400,000 uncased vocabulary items. The dimensionality of GloVe in our model is 50 dimensions. The outputs of this layer are two matrices, $L_{E}\in\mathbb{R}^{S_{e}\times W_{e}\times d_{L}}$ for the essay and $L_{A}\in\mathbb{R}^{S_{a}\times W_{a}\times d_{L}}$ for the article, where $S_{e}$, $S_{a}$, $W_{e}$, $W_{a}$, and $d_{L}$ are number of sentences of the essay and the article, length of sentences of the essay and the article, and the embedding size, respectively. Same to Dong et al. (2017), a dropout is applied after the word embedding layer.

4.2 Word Level Convolutional Layer

In this layer, we perform 1D convolution over the word representations of both $L_{E}$ and $L_{A}$, so that we can get local representation of each sentence. For each word $w_{i}$ in each sentence, we perform 1D convolution:

[TABLE]

where $g$ is a nonlinear activation, $k$ is the kernel size, $U_{p}$ is the filter weight matrix, and $b_{p}$ is the bias vector. The outputs of this layer are $C_{e}\in\mathbb{R}^{S_{e}\times P_{e}\times d_{C}}$ for the essay and $C_{a}\in\mathbb{R}^{S_{a}\times P_{a}\times d_{C}}$ for the article, where $P_{e}$ and $P_{a}$ are filtered lengths of sentences of the essay and the article, respectively. $d_{C}$ is the number of filters of the 1D convolution layer.

4.3 Word Level Attention Pooling Layer

After the convolutional layer, a pooling layer is demanded to obtain the sentence representations. In this layer, we follow the same design presented by Dong et al. (2017). The attention pooling is defined as equations below:

[TABLE]

[TABLE]

[TABLE]

where $U_{m}$, $u_{v}$ and $b_{m}$ are weight matrix, vector, and bias vector, respectively. $m_{i}$ and $v_{i}$ are attention vector and attention weight for $p_{i}$. The outputs of this layer are $A_{e}\in\mathbb{R}^{S_{e}\times d_{C}}$ for the essay and $A_{a}\in\mathbb{R}^{S_{a}\times d_{C}}$ for the article.

4.4 Sentence Level LSTM Layer

In this layer, we use a Long Short-Term Memory Network (LSTM) (Hochreiter and Schmidhuber, 1997) over the sentence representations of the essay and the article to capture contextual evidence from previous sentences to refine the sentence representation.

The LSTM unit is a special kind of RNN unit which has long-term dependency learning ability. LSTMs use three gates to control information flow to avoid the long-term dependency problem by forgetting or remembering information in each LSTM unit. They are an input gate, a forget gate, and an output gate. The following equations define the LSTM unit:

[TABLE]

[TABLE]

[TABLE]

[TABLE]

[TABLE]

[TABLE]

where $s_{t}$ and $h_{t}$ are the input sentence and the output state of time $t$, respectively. $W_{f}$, $W_{i}$, $W_{c}$, and $W_{o}$ are weight matrices. $b_{f}$, $b_{i}$, $b_{c}$, and $b_{o}$ are bias vectors. $\sigma$ is the sigmoid function, and $*$ is element-wise multiplication. The output of this layer are $H_{e}\in\mathbb{R}^{S_{e}\times d_{H}}$ for the essay and $H_{a}\in\mathbb{R}^{S_{a}\times d_{H}}$ for the article, where $d_{H}$ is the dimensionality of the output.

4.5 Sentence Level Co-Attention Layer

The concept of this layer is presented by Seo et al. (2017) in the part of attention flow layer. This layer links information from $H_{e}$ and $H_{a}$, and generates a collection of article aware features vector of essay sentences. The attention is computed in two directions, from essay to article, and vice versa. Both attention scores are figured from a similarity matrix by the following equation:

[TABLE]

where $W_{sim}$ is weight matrix, $he_{t}$ and $ha_{j}$ are $t_{th}$ row vector of $H_{e}$ and $j_{th}$ row vector of $H_{a}$, $b_{sim}$ is bias vector. $*$ is element-wise multiplication. $[;]$ is vector concatenation. After obtaining the similarity matrix $Sim\in\mathbb{R}^{S_{e}\times S_{a}}$, we compute the attention in two directions.

Essay to Article Attention measures which sentences in the article are similar to each sentence in students’ essays. The following equations define the essay to article attention:

[TABLE]

[TABLE]

where $a_{ea}\in\mathbb{R}^{S_{e}\times S_{a}}$ represents the attention score of each sentence in the article associate with each sentence in the essay, $softmax$ is performed across each row. The output of this $\tilde{H}_{a}\in\mathbb{R}^{S_{e}\times d_{H}}$.

Article to Essay Attention measures which sentences in the essay have the closest meaning to one of the sentences in the article. The following equations define the article to essay attention:

[TABLE]

[TABLE]

where $a_{ae}\in\mathbb{R}^{S_{e}}$, $max_{col}$ is a maximum function performed across the column, and $\tilde{h}_{e}\in\mathbb{R}^{d_{H}}$. Because $max_{col}$ will find out which sentence in the article has the closest meaning to each sentence in the essay, so $\tilde{h}_{e}$ represents the attention score of the most important sentence in the essay associated with the article. After tiling $S_{e}$ times, the final output of this layer is $\tilde{H}_{e}\in\mathbb{R}^{S_{e}\times d_{H}}$.

The final output $G$ is a concatenated matrix of $H_{e}$, $\tilde{H}_{e}$, and $\tilde{H}_{a}$ defined by:

[TABLE]

where $*$ is element-wise multiplication, and $[;]$ is concatenation, $H_{e}$ is the original representation of essay, $\tilde{H}_{a}$ is the essay to article attention, $H_{e}*\tilde{H}_{a}$ is the self-aware representation, and $H_{e}*\tilde{H}_{e}$ is article-aware representation. Therefore, the output of this layer is $G\in\mathbb{R}^{S_{e}\times 4d_{H}}$, the article-aware representation of each sentence in the essay.

4.6 Modeling Layer

$G$ is the representation of each sentence, and we need the representation of the essay. Therefore, we introduce another LSTM layer for modeling the essay and only use the output of the final LSTM unit as the output of this layer $M\in\mathbb{R}^{d_{M}}$, where $d_{M}$ is the dimensionality of the output of LSTM units.

4.7 Output Layer

After obtaining the essay representation $M$, a linear layer with sigmoid activation will predict the final output. The following equation defines the output layer:

[TABLE]

where $W_{o}$ is weight vector, and $b_{o}$ is bias vector. $y$ is the final predicted score of the essay.

5 Training

Loss. Dong et al. (2017) used mean squared error (MSE) loss, thus we use the same loss function. MSE evaluates the average of squared error between the predicted score and the gold standard. Thus it is widely used in regression tasks. The following equation defines MSE:

[TABLE]

where $y_{i}$ is the predicted score, $y^{\prime}_{i}$ is the gold standard, $N$ it the total number of samples.

Optimization. The optimizer we use is RMSprop (Dauphin et al., 2015). The initial learning rate is 0.001, momentum is 0.9, and Dropout rate is 0.5 for preventing overfitting. These setting are the same as used by Dong et al. (2017).

6 Experimental Setup

We configure experiments to test three hypotheses:

• H1:

the model we proposed (denoted by CO-ATTN) will outperform or at least perform equally well as the baseline (denoted by SELF-ATTN) presented by Dong et al. (2017) on four ASAP essay corpora in the holistic score prediction task.

• H2:

the model we proposed will outperform or at least perform equally well as the baseline on two RTA corpora in the Evidence score prediction task.

• H3:

the model we proposed will outperform or at least perform equally well as the non-neural network baselines on both corpora.

We use NLTK (Bird et al., 2009) for text preprocessing. The vocabulary size of the data is limited to 4000, and all scores are scaled to the range [0, 1], following Taghipour and Ng (2016) and Dong et al. (2017). In particular, the 4000 most frequent words are preserved, with all other words treated as unknowns. The assessment scores will be converted back to their original range during evaluation. We use Quadratic Weighted Kappa (QWK) to evaluate our model. QWK is not only the official criteria of ASAP corpus, but also adopted as evaluation metric in Rahimi et al. (2014); Taghipour and Ng (2016); Dong et al. (2017); Rahimi et al. (2017); Zhang and Litman (2017) for both ASAP and RTA corpora.

We use 5-fold cross-validation because both RTA and ASAP corpora have no released labeled test data. We split all corpora into 5 folds. For the ASAP corpus, the partition is the same as the setting presented by Taghipour and Ng (2016). For the RTA corpus, since there is no prior work to split the corpus, we separate it into 5 folds randomly. In each fold, 60% of the data are used for training, 20% of the data are the development set, and 20% of the data are used for testing.

To select the best model, we trained each model on 100 epochs and evaluated on the development set after each epoch. The best model is the model with the best QWK on the development set. This is done five times, once for each partition in the cross-validation. Then the average QWK score from these five evaluations on the test set is reported. Paired t-tests are used for significance tests with $p<0.05$. Table 3 shows all hyper-parameters for training.

The code of SELF-ATTN are provided by Dong et al. (2017), they used Keras (Chollet et al., 2015) 1.1.1 and Theano (Theano Development Team, 2016) 0.8.2 as the backend. Because we are using Keras 2.1.3 and TensorFlow (Abadi et al., 2015) 1.4.0 as the backend, we ran all experiments with our frameworks. Therefore, the numbers of SELF-ATTN have small differences to the numbers reported by the baseline model.

For non-neural network baselines, we introduce the SVR and BLRR baselines presented by Phandi et al. (2015) for the ASAP corpus, and SG baseline presented by Zhang and Litman (2017) for the RTA corpus.

SVR and BLRR models use Enhanced AI Scoring Engine (EASE)222https://github.com/edx/ease to extract four types of features, such as length, part of speech, prompt, and the bag of words. Then they use SVR and BLRR as the classifiers, respectively. We do not perform any significance test on both SVR and BLRR because we do not have detailed experiment data. Therefore, we only report the result presented in Phandi et al. (2015).

SG model extracts evidence features based on hand-crafted topic and example lists, and uses random forest tree as the classifier. We follow the same data partition. However, we only use the training set for training and the testing set for testing while ignoring the development set so that we can perform the same paired t-tests in the experiments.

7 Results

We first examine H1. The results shown in Table 4 support this hypothesis. The CO-ATTN model yields higher performance than the SELF-ATTN model on all ASAP prompts. However, the CO-ATTN model only significantly outperforms the SELF-ATTN model on Prompt 3.

Second, we examine H2. Again, the results shown in Table 4 support this hypothesis. The CO-ATTN model yields higher performance than the SELF-ATTN model, significantly on both of the RTA corpora.

Last, we examine H3. The results shown in Table 4 still support this hypothesis. The CO-ATTN model yields higher performance than all non-neural network baselines.

The results show that in our tasks, the neural network approaches are better than non-neural network baselines. One possible reason is the final representation of the essay from neural network contains more information. However, some of the information might be ignored by hand-crafted features. For example, the importance of different evidence in RTA task is not considered in the SG model. It treats all evidence equally. However, the neural network models capture this information automatically.

Apparently, the CO-ATTN model performs better in the RTA tasks, because it always significantly outperforms the SELF-ATTN model. One possible reason is that the RTA task only considers the Evidence score. The CO-ATTN model is more suitable for the Evidence score prediction task because it can find pieces of evidence that appear in both students’ essays and the source article better. In contrast, the SELF-ATTN model only considers students’ essays associated with the scores. In this case, if a piece of evidence is not mentioned by students, this data-driven model cannot distinguish it. Consequently, some important pieces of evidence will be assigned to a lower weight. However, the CO-ATTN model considers not only the students’ essays but also the source article. In other words, if an important piece of evidence is not mentioned by too many students, but it is in the source article, the CO-ATTN model will assign this sentence higher attention.

In the ASAP holistic score prediction task, although we still see a benefit in using the CO-ATTN model, it is reduced. In this case, the benefit we saw in the Evidence dimension from the CO-ATTN model becomes less significant because the model also needs to consider more aspects of the essay, such as organization, grammar mistakes, and so on. Our results suggest that the co-attention mechanism of the CO-ATTN model cannot capture these aspects significantly better than the SELF-ATTN model. Therefore, the CO-ATTN model only significantly outperforms the SELF-ATTN model on Prompt 3.

8 Discussion

In Table 5, we list 10 sentences from student $RTA_{MVP}$ essays and their associated attention scores. Because we have a list of examples manually extracted by our experts as important evidence from the $RTA_{MVP}$ source article, examining RTA data helps us understand the attention score assigned by our model. Bolded are examples extracted by the expert from the source article that the student includes in the essay. A lower attention score means this sentence is less important. Otherwise, the score is high. As we can see, sentences 1, 2, 3, and 4 are low attention sentences, sentences 5, 6, and 7 are mid attention sentences, and sentences 8, 9, and 10 are high attention sentences. The attention scores reflect the importance of these sentences accurately.

Sentence 1 is a short and general sentence related to the source article, but it has no specific evidence from it. Sentence 2 even has no content related to the source article. Sentence 3 has many details related to the source article. However, it still has no evidence directly from the source article. Sentence 4 mentions “The author did convince me that winning the fight against poverty is achievable in our lifetime” which comes from both the prompt and the source article, but this statement is so general that almost every student mentions this statement in the essay which makes this statement not distinguishable. For these reasons, these four sentences receive low attention scores.

Although sentence 5 is short, it mentions one piece of evidence. Sentence 6 talks about farming which is a topic from the source article. In the article, the things listed in this sentence are things the farmer needs to worry about. However, this sentence indicates “the farmer don’t have to worry” because of the MVP project. Sentence 7 also mentions conditions of hospitals nowadays. However, it mentions not only water but also electricity which is more than Sentence 5. For these reasons, these three sentences receive mid attention scores from low to high.

The last three sentences receive high attention scores because they all use more pieces of evidence directly from the source article. Sentence 8 talks about the school, and Sentence 9 talks about the hospital. Sentence 10 talks about farming. However, sentence 10 receives the highest attention score, because it mentions evidence from both before and after the MVP project.

From these sentences, we can also see that the attention score depends on neither the length of the sentence nor only the specificity of the sentence. It instead depends on how many important pieces of evidence there are in the sentence. For example, Sentence 3 is long and talks about some details of our modern life. Although it also talks about quality materials or better housing and clothing compared to people living in Kenya, it receives a low attention score because there is no specific evidence directly from the source article. In contrast, Sentence 9 is shorter than Sentence 3. However, it receives a higher attention score because it mentions many pieces of evidence from the source article.

Overall, the CO-ATTN model seems to capture the importance of sentences by assigning reasonable attention scores based on the relevance of the sentence to the source article.

9 Conclusion and Future Work

In this paper, we presented a co-attention based neural network model that outperforms a state of the art attention based neural network model for essay scoring, not only for RTA Evidence assessment but also for holistic assessment of ASAP source-dependent responses. Advantages of our model are that it does not need any expert preprocessing of the source article; the input of this model is only the raw student essay and its source article. Moreover, our model somewhat captures the importance of different pieces of evidence, although it is not specifically designed for this purpose. However, quantitative experiments that can answer whether the attention scores are correlated to the importance of different pieces of evidence need to be done. Also, this leads to an interesting future investigation, development of a neural network approach that both has an acceptable score prediction, and can simultaneously generate evidence lists from the source article. Another interesting future investigation could be examining the ability of this model to generalize to a new prompt.

Acknowledgments

We would like to show our appreciation to Tazin Afrin and Luca Lugini for their comments on an earlier version of the paper. We are also immensely grateful to every member of the RTA group for sharing their pearls of wisdom with us.

The research reported here was supported, in whole or in part, by the Institute of Education Sciences, U.S. Department of Education, through Grant R305A160245 to the University of Pittsburgh. The opinions expressed are those of the authors and do not represent the views of the Institute or the U.S. Department of Education.