Knowledge Query Network: How Knowledge Interacts with Skills

TL;DR

The paper introduces Knowledge Query Network (KQN), a neural network model for Knowledge Tracing that improves prediction accuracy and offers intuitive interpretability of knowledge states and skill interactions.

Contribution

KQN models knowledge and skill interactions with neural networks, introducing probabilistic skill similarity for better interpretability and domain analysis.

Findings

KQN outperforms existing models in prediction accuracy.

Knowledge-skill interactions can be visualized for interpretation.

Skill domain analysis via clustering using KQN's skill vectors.

Abstract

Knowledge Tracing (KT) is to trace the knowledge of students as they solve a sequence of problems represented by their related skills. This involves abstract concepts of students' states of knowledge and the interactions between those states and skills. Therefore, a KT model is designed to predict whether students will give correct answers and to describe such abstract concepts. However, existing methods either give relatively low prediction accuracy or fail to explain those concepts intuitively. In this paper, we propose a new model called Knowledge Query Network (KQN) to solve these problems. KQN uses neural networks to encode student learning activities into knowledge state and skill vectors, and models the interactions between the two types of vectors with the dot product. Through this, we introduce a novel concept called \textit{probabilistic skill similarity} that relates the pairwise cosine and Euclidean distances between skill vectors to the odds ratios of the corresponding skills, which makes KQN interpretable and intuitive. On four public datasets, we have carried out experiments to show the following: 1. KQN outperforms all the existing KT models based on prediction accuracy. 2. The interaction between the knowledge state and skills can be visualized for interpretation. 3. Based on probabilistic skill similarity, a skill domain can be analyzed with clustering using the distances between the skill vectors of KQN. 4. For different values of the vector space dimensionality, KQN consistently exhibits high prediction accuracy and a strong positive correlation between the distance matrices of the skill vectors.

Tables

Table 1. (a) Original

T	SID	C
1	1	0
2	2	0
3	1	1

Table 2. (a) Original

T	SID	C
1	1	0
2	2	0
3	1	1

Table 3. (b) BKT

T	SID	C
1	1	0
2	1	1

Table 4. (c) LFA

T	SID	NO1	NO2
1	1	1	0
2	2	1	1
3	1	2	1

Table 5. (d) PFA

T	SID	S1	F1	S2	F2
1	1	0	1	0	0
2	2	0	1	0	1
3	1	1	1	0	1

Table 6. Table 2 . Statistics for all the datasets. Names of the datasets have been abbreviated. ‘Size’ and ‘Max Steps’ refer to the total number of student responses and the maximum number of time steps, respectively.

Dataset	Students	Skills	Size	Max Steps
ASSIST2009	4,151	110	325,637	1,261
ASSIST2015	19,840	100	683,801	618
Statics2011	333	1,223	189,297	1,181
Synthetic-5	4,000	50	200,000	50

Table 7. Table 3 . Accuracy of different KT models were compared based on test AUCs (%) for the correctness prediction task. IRT, BKT, and their variants were used as representatives of non-neural-network KT models. DKVMN and DKT were compared to KQN as baselines of neural-network models while DKVMN has been the previous state-of-the-art neural network KT model. Note that DKT+KQN refers to DKT with the embedded skill vectors learned from KQN.

Dataset	Test AUC (%)
	IRT+	BKT+	DKVMN	DKT	DKT+KQN	KQN
ASSIST2009	77.40	-	81.57$\pm$0.1	80.53$\pm$0.2	82.05$\pm$0.04	82.32$\pm$0.05
ASSIST2015	-	-	72.68$\pm$0.1	72.52$\pm$0.1	73.41$\pm$0.02	73.40$\pm$0.02
Statics2011	-	75	82.84$\pm$0.1	80.20$\pm$0.2	80.27$\pm$0.22	83.20$\pm$0.05
Synthetic-5	-	80	82.73$\pm$0.1	80.34$\pm$0.1	82.58$\pm$0.01	82.81$\pm$0.01

Table 8. Table 4 . 14 flat clusters of ASSISTments 2009-2010 skills based on the average linkage method and the Euclidean distance. In each cluster, skills are sorted in an ascending order based on skill IDs. Different clusters are separated by dashed lines.

2 Area Irregular Figure	41 Finding Percents	74 Multiplication and Division Positive Decimals	55 Divisibility Rules
46 Algebraic Solving	42 Pattern Finding	107 Parts of a Polynomial, Terms, Coefficient, Monomial, Exponent, Variable	57 Perimeter of a Polygon
\cdashline3-3 56 Reading a Ruler or Scale	43 Write Linear Equation from Situation	4 Table	71 Angles on Parallel Lines Cut by a Transversal
63 Scale Factor	44 Square Root	12 Circle Graph	72 Write Linear Equation from Ordered Pairs
67 Percents	47 Percent Discount	32 Box and Whisker	80 Unit Conversion Within a System
78 Rate	54 Interior Angles Triangle	49 Complementary and Supplementary Angles	83 Area Parallelogram
84 Effect of Changing Dimensions of a Shape Proportionally	62 Ordering Real Numbers	53 Interior Angles Figures with More than 3 Sides	91 Polynomial Factors
85 Surface Area Cylinder	65 Scientific Notation	58 Solving for a variable	97 Choose an Equation from Given Information
86 Volume Cylinder	76 Computation with Real Numbers	59 Exponents	101 Angles - Obtuse, Acute, and Right
88 Solving Systems of Linear Equations	79 Solving Inequalities	68 Area Circle	104 Simplifying Expressions positive exponents
\cdashline4-4 92 Rotations	81 Area Rectangle	70 Equation Solving More Than Two Steps	20 Addition and Subtraction Integers
93 Reflection	82 Area Triangle	75 Volume Sphere	31 Circumference
96 Interpreting Coordinate Graphs	87 Greatest Common Factor	102 Distributive Property	34 Conversion of Fraction Decimals Percents
\cdashline1-1\cdashline3-3 14 Proportion	89 Solving Systems of Linear Equations by Graphing	1 Area Trapezoid	60 Division Fractions
66 Write Linear Equation from Graph	90 Multiplication Whole Numbers	6 Stem and Leaf Plot	77 Number Line
\cdashline1-1\cdashline4-4 69 Least Common Multiple	98 Intercept	10 Venn Diagram	19 Multiplication Fractions
\cdashline1-1 3 Probability of Two Distinct Events	99 Linear Equations	11 Histogram as Table or Graph	25 Subtraction Whole Numbers
5 Median	100 Slope	21 Multiplication and Division Integers	61 Estimation
7 Mode	105 Finding Slope from Ordered Pairs	22 Addition Whole Numbers	109 Finding Slope From Equation
\cdashline4-4 8 Mean	106 Finding Slope From Situation	26 Equation Solving Two or Fewer Steps	24 Addition and Subtraction Fractions
9 Range	108 Recognize Quadratic Pattern	33 Ordering Integers	50 Pythagorean Theorem
13 Equivalent Fractions	110 Quadratic Formula to Solve Quadratic Equation	37 Ordering Positive Decimals	103 Recognize Linear Pattern
\cdashline2-2\cdashline4-4 15 Fraction Of	16 Probability of a Single Event	38 Rounding	29 Counting Methods
18 Addition and Subtraction Positive Decimals	45 Algebraic Simplification	39 Volume Rectangular Prism	95 Midpoint
\cdashline4-4 23 Absolute Value	73 Prime Number	40 Order of Operations All	64 Surface Area Rectangular Prism
\cdashline4-4 28 Calculations with Similar Figures	94 Translations	48 Nets of 3D Figures	36 Unit Rate
\cdashline2-2 30 Ordering Fractions				17 Scatter Plot	51 D.4.8-understanding-concept-of-probabilities
35 Percent Of	27 Order of Operations +,-,/,* () positive reals	52 Congruence

Table 9. Table 5 . Average ARI scores for different linkage methods and distance measures on Synthetic-5.

Linkage	Distance	ARI
average	cosine	0.3180
	Euclidean	0.3266
centroid	cosine	0.0373
	Euclidean	0.0143
complete	cosine	0.2898
	Euclidean	0.2898
median	cosine	0.0368
	Euclidean	0.0071
single	cosine	0.0703
	Euclidean	0.0703
ward	cosine	0.3201
	Euclidean	0.3234
weighted	cosine	0.2996
	Euclidean	0.3020

Table 10. Table 6 . Test AUCs (%) of DKT on ASSISTments 2009-2010 after replacing skill IDs with cluster labels assigned by flat clustering. The average linkage and the Euclidean distance were used.

Number of Clusters	Test AUC (%)
5	79.77$\pm$0.03
6	79.97$\pm$0.03
7	80.10$\pm$0.03
8	80.04$\pm$0.02
9	80.14$\pm$0.03
10	80.10$\pm$0.02
11	80.23$\pm$0.05
12	80.20$\pm$0.05
13	80.55$\pm$0.03
14	80.64$\pm$0.03

Table 11. Table 7 . Average pairwise distances and average differences between the pairwise distances.

Distance	Dataset	${\xi }_{d_{opt},0.5d_{opt}}$	${\xi }_{d_{opt},2d_{opt}}$	${\xi }_{0.5d_{opt},2d_{opt}}$	${\eta }_{d_{opt}}$	${\eta }_{0.5d_{opt}}$	${\eta }_{2d_{opt}}$
cosine	ASSIST2009	0.11	0.09	0.10	0.75	0.76	0.76
	ASSIST2015	0.11	0.09	0.10	0.70	0.69	0.69
	Statics2011	0.15	0.09	0.20	0.44	0.55	0.37
	Synthetic-5	0.12	0.10	0.11	0.78	0.78	0.79
Euclidean	ASSIST2009	0.09	0.07	0.09	1.22	1.23	1.23
	ASSIST2015	0.09	0.07	0.09	1.17	1.17	1.17
	Statics2011	0.16	0.11	0.21	0.92	1.04	0.84
	Synthetic-5	0.10	0.08	0.09	1.25	1.24	1.25

Table 12. Table 8 . Test AUCs of KQNs by varying the embedding dimensionality d 𝑑 d . d o ​ p ​ t subscript 𝑑 𝑜 𝑝 𝑡 d_{opt} refers to the optimal values found from the correctness prediction task. Note that prediction accuracy may not be the highest when d 𝑑 d was set to d o ​ p ​ t subscript 𝑑 𝑜 𝑝 𝑡 d_{opt} .

Dataset	Test AUC (%)
	$d_{opt}$	$0.5d_{opt}$	$2d_{opt}$
ASSIST2009	82.32	82.35	82.32
ASSIST2015	73.40	73.38	73.40
Statics2011	83.20	83.17	83.16
Synthetic-5	82.81	82.79	82.82

Table 13. Table 9 . Mantel tests on the distance matrices. p-values are not marked since they were 0.001 in all cases.

Distance	Dataset	$\rho$
		$d_{opt}$-$0.5d_{opt}$	$d_{opt}$-$2d_{opt}$	$0.5d_{opt}$-$2d_{opt}$
cosine	ASSIST2009	0.521	0.653	0.570
	ASSIST2015	0.582	0.661	0.609
	Statics2011	0.616	0.825	0.609
	Synthetic-5	0.495	0.526	0.511
Euclidean	ASSIST2009	0.531	0.660	0.583
	ASSIST2015	0.601	0.682	0.629
	Statics2011	0.620	0.816	0.607
	Synthetic-5	0.508	0.536	0.523

Equations

$$p_{t+1}$$

$$c_{t+1}$$

$${\mathbf{x}}_t\in \{0,1{\}}^{2N},$$

$${\mathbf{x}}_t^k=1\text{ if the answer is wrong},$$

$${\mathbf{x}}_t^{k+N}=1\text{ if the answer is correct},$$

$$y_{t+1}={\mathbf{KS}}_t\cdot {\mathbf{s}}_{t+1},$$

$$p_{t+1}=\sigma (y_{t+1}),$$

$$P(c_{t+1}=1|e_{1:t+1},c_{1:t})$$

$$\displaystyle\approx\sigma(y_{t+1})=\sigma\bigl{(}\mathbf{KS}_{t}\cdot\mathbf{s}_{t+1}\bigr{)}.$$

$${\mathbf{h}}_t=RNN({\mathbf{x}}_t),$$

$${\mathbf{KS}}_t={\mathbf{W}}_{h,KS}\cdot {\mathbf{h}}_t+{\mathbf{b}}_{h,KS},$$

$${\mathbf{o}}_{t+1}=ReLU({\mathbf{W}}_1\cdot (ReLU({\mathbf{W}}_0\cdot {\mathbf{e}}_{t+1}+{\mathbf{b}}_0))+{\mathbf{b}}_1),$$

$${\mathbf{s}}_{t+1}=L2\text{-}normalize({\mathbf{o}}_{t+1}),$$

$${\mathbf{s}}_{t+1}\in {\mathbf{U}}^d=\{\mathbf{v}\in {\mathbb{R}}^d\text{ : }||\mathbf{v}||=1,{\mathbf{v}}^i\ge 0\text{ for }i=1,\cdots ,d\},$$

$$E({\mathbf{\theta }}_{model}|c_{t+1},p_{t+1})$$

$$\displaystyle=-\biggl{[}c_{t+1}\log p_{t+1}+(1-c_{t+1})\log(1-p_{t+1})\biggr{]},$$

$$E_{total}({\mathbf{\theta }}_{model}|c_{2:t+1},p_{2:t+1})$$

$$=\sum _{t=1}^{T-1}E({\mathbf{\theta }}_{model}|c_{t+1},p_{t+1}).$$

$$d_{Euclidean}({\mathbf{s}}_1,{\mathbf{s}}_2{)}^2=||{\mathbf{s}}_1-{\mathbf{s}}_2|{|}^2=2(1-{\mathbf{s}}_1\cdot {\mathbf{s}}_2)$$

$$=2d_{cosine}({\mathbf{s}}_1,{\mathbf{s}}_2),\forall {\mathbf{s}}_1,{\mathbf{s}}_2\in {\mathbf{U}}^d.$$

$$p=P(c=1|\mathbf{KS},\mathbf{s}),o=\frac{p}{1-p}.$$

$$\displaystyle\biggl{(}\log\frac{o_{1}}{o_{2}}\biggr{)}^{2}$$

$$=(y_1-y_2{)}^2$$

$$=(\mathbf{KS}\cdot {\mathbf{s}}_1-\mathbf{KS}\cdot {\mathbf{s}}_2{)}^2$$

$$\displaystyle=\bigl{(}\mathbf{KS}\cdot(\mathbf{s}_{1}-\mathbf{s}_{2})\bigr{)}^{2}$$

$$\displaystyle=\bigl{(}\mathbf{KS}\cdot\boldsymbol{\Delta}_{1,2}\bigr{)}^{2}\times||\mathbf{s}_{1}-\mathbf{s}_{2}||^{2}$$

$$\displaystyle=\bigl{(}\mathbf{KS}\cdot\boldsymbol{\Delta}_{1,2}\bigr{)}^{2}\times d_{Euclidean}(\mathbf{s}_{1},\mathbf{s}_{2})^{2}$$

$$\displaystyle=\bigl{(}\mathbf{KS}\cdot\boldsymbol{\Delta}_{1,2}\bigr{)}^{2}\times 2d_{cosine}(\mathbf{s}_{1},\mathbf{s}_{2}),$$

$${}^{\forall }{d}_1,d_2\in \{d_{opt},0.5d_{opt},2d_{opt}\},d_1\ne d_2,$$

$${\xi }_{d_1,d_2}=\frac{{\sum }_{i>j}|{pdist}_{d_1}({\mathbf{s}}_i,{\mathbf{s}}_j)-{pdist}_{d_2}({\mathbf{s}}_i,{\mathbf{s}}_j)|}{\left(\genfrac{}{}{0ex}{}{N}{2}\right)},$$

$${\eta }_d=\frac{{\sum }_{i>j}^N{pdist}_d({\mathbf{s}}_i,{\mathbf{s}}_j)}{\left(\genfrac{}{}{0ex}{}{N}{2}\right)}.$$

Full text

On four public datasets, we have carried out experiments to show the following: 1. KQN outperforms all the existing KT models based on prediction accuracy. 2. The interaction between the knowledge state and skills can be visualized for interpretation. 3. Based on probabilistic skill similarity, a skill domain can be analyzed with clustering using the distances between the skill vectors of KQN. 4. For different values of the vector space dimensionality, KQN consistently exhibits high prediction accuracy and a strong positive correlation between the distance matrices of the skill vectors.

Knowledge Tracing, Deep Learning, Learning Analytics, Educational Data Mining, Massive Open Online Courses, Intelligent Tutoring Systems, Learner Modeling, Knowledge Modeling, Domain Modeling

††journalyear: 2019††copyright: acmcopyright††conference: The 9th International Learning Analytics & Knowledge Conference; March 4–8, 2019; Tempe, AZ, USA††booktitle: The 9th International Learning Analytics & Knowledge Conference (LAK19), March 4–8, 2019, Tempe, AZ, USA††price: 15.00††doi: 10.1145/3303772.3303786††isbn: 978-1-4503-6256-6/19/03††ccs: Computing methodologies Neural networks††ccs: Applied computing E-learning

1. Introduction

One of the advantages of an Intelligent Tutoring System (Nwana, 1990) and massive open online courses (Kaplan and Haenlein, 2016) is that they can potentially benefit from monitoring and tracking student activities in an adaptive learning environment, where learner modeling comes into play. A learner model provides estimates for the students’ state, and includes two inter-connected aspects: domain modeling and knowledge modeling (Pelánek, 2017).

A domain model (Pelánek, 2017) studies the structure within a domain of problems (For example, it finds out which skill a problem is related to: “1+2=?” to “addition of integers” and “1.3+2.5=?” to “addition of decimals”). Another task of a domain model is to discover the structure of a skill domain, which can be performed either manually or automatically (Pelánek, 2017). On the other hand, a knowledge model (Anderson et al., 1990), in an abstract sense, traces students’ knowledge while they are solving problems. Knowledge has been described in various forms by the name of knowledge state, which has no universal definition yet. In this paper, the term knowledge state has been used as a state that can describe a student’s general level of attainment of skills. Often, domain modeling and knowledge modeling are viewed to be separate; however, we tried to provide approaches for both where problem-solving records of students can be important input features for finding the latent structure of a skill domain.

KT is a research area which analyzes student activities and studies knowledge acquisition, where its main task is to describe a student’s knowledge. To elaborate, consider a student who solved a sequence of problems. Then the student’s data is given by the temporal sequence of tuples, each of which is consisted of the skill that the problem at each time step is related to and the binary correctness that indicates whether or not the student gave a right answer. By calling such tuple student response, the KT problem is formulated as follows: 1. given the student responses up to the time step $t$, describe the student’s knowledge state at the current time step $t$, and 2. given the skill at the next time step $t+1$, predict the correctness by modeling the interaction between the student’s knowledge state at time $t$ and the skill at time $t+1$, which we will call knowledge interaction. Note that the knowledge state refers to the dynamic state of a student accumulated from the student responses while a skill indicates a particular ability that needs to be learned by a student to solve a problem.

Therefore, the quality of a KT model is measured by its ability to describe the knowledge state of a student and its accuracy of predicting correctness. Additionally, since modeling the knowledge interaction is to describe how a student’s knowledge state responds to different skills, it is desirable if a KT model can explain the relationship between skills that can be inferred from the knowledge interaction. For example, we can say that “addition of integers” is independent of “subtraction of integers” if a model observes that a student does not learn the latter while learning the former. Similarly, they are dependent if the change in a student’s knowledge state of one skill affects the knowledge state of the other. We believe that modeling such skill relationship can lead to further exploration of the latent structure of the skill domain, which is the subject of domain modeling.

However, existing KT models provide limited definitions of either knowledge state, or knowledge interaction, or both. For example, Bayesian Knowledge Tracing (BKT) (Anderson et al., 1990) imposes a binary assumption on the knowledge state, which is too restrictive to be intuitive, and Deep Knowledge Tracing (DKT) (Piech et al., 2015) does not give an explanation of knowledge interaction. In this paper, we propose a new neural network KT model called Knowledge Query Network (KQN) to generalize the knowledge state and explain the knowledge interaction more descriptively. The central idea is to use the dot product between a knowledge state vector and a skill vector to define the knowledge interaction while leveraging neural networks to encode student responses and skills into vectors of the same $d$ dimensionality. Additionally, we introduce a novel concept called probabilistic skill similarity which relates the cosine and Euclidean distances between the skill vectors to the odds ratios for the corresponding skills. Based on those distances, we explore the latent structure of a skill domain with cluster analysis. Lastly, we show that KQN is stable in predicting correctness and learning skill vectors by comparing prediction accuracy and the distance matrices of the skill vectors when the vector space dimensionality is varied.

2. Related Work

Item Response Theory (IRT) is a framework for modeling the relationship between problems and correctness (Hulin et al., 1983). In its simplest form, it uses a logistic regression model by estimating student proficiency and skill difficulty. However, it assumes the proficiency to be constant and does not explain any structure for problems. To overcome those limitations, Bayesian extensions of IRT have been proposed to have a hierarchical structure over items (HIRT) and temporal changes in a student’s knowledge state (TIRT) (Wilson et al., 2016). Still, HIRT assumes constant student proficiency while TIRT lacks the ability of domain analysis.

In BKT, a student’s knowledge state is viewed as a set of binary latent variables, one for each skill, with two possible states, known and unknown (Anderson et al., 1990). Then a set of observable variables, each of which corresponds to correctness per skill, are conditioned on the set of the binary variables. For example, let us say we have a running example of student Ben throughout this paper, who has records as shown in Table 1. Accordingly, there will be two knowledge states and correctness variables for skills 1 and 2, a total of four variables with two independent BKT models with input data as shown in Table 1(b). Then the knowledge acquisition in BKT is modeled with a Hidden Markov model (HMM), where knowledge interaction is controlled by a set of interpretable equations. Since BKT lacks the ability to forget and individualize, a number of extensions have been proposed (Yudelson et al., 2013; Khajah et al., 2016). However, most importantly, BKT’s independence assumption on different skills is considered to be highly constrained and not effective, where the model cannot leverage the whole data. For example, Ben’s history of responses on skill 1 cannot tell his responses on skill 2 since there should be two separate models for each skill.

Learning Factors Analysis (LFA) (Cen et al., 2006) models a student’s knowledge state as a set of binary variables, one for each skill. Correctness at each time step is predicted with a logistic regression model which has covariates related to students, skills, and a summary statistic of student responses, i.e., the number of past opportunities for each skill. Performance Factors Analysis (PFA) (Pavlik et al., 2009) extends LFA by separating ‘the number of opportunities per skill’ into ‘the number of correct answers per skill’ and ‘the number of incorrect answers per skill’ with the others the same, e.g., in Ben’s case, the input data are preprocessed as shown in Table 1(c) and Table 1(d).

Since an estimate for correctness is explained with student covariates, and skill-specific covariates without variable interactions in LFA and PFA, the two models do not describe how a student’s knowledge state with respect to one skill is affected by that with respect to another skill; instead, a student parameter, which is also called student proficiency, is the only factor that relates the knowledge state for different skills. Moreover, a skill is explained by the regression coefficients for the skill-specific covariates from which we cannot tell the structure of a skill domain directly.

As the first neural network KT model, DKT (Piech et al., 2015), given the student’s responses as input, encodes a student’s knowledge state as a summarized vector calculated from a recurrent neural network (RNN), which is a renowned neural network technique for modeling temporal data. However, DKT does not define the knowledge interaction directly. In detail, a student response at each time step is formed as a tuple $(q_{t},a_{t})$, where $q_{t}$ and $a_{t}$ refer to the problem ID and correctness, respectively. For example, Ben’s original data in Table 1(a) is expressed as $(1,0)$ at $t=1$. The input is then passed to an RNN layer, where Long Short-Term Memory (LSTM) (Hochreiter and Schmidhuber, 1997) was used in the original paper (Piech et al., 2015), and its output hidden state is passed to a logistic function after an affine transformation, i.e., $\mathbf{y}_{t}=\sigma(\mathbf{W}\cdot\mathbf{h}_{t}+\mathbf{b})$, where $\mathbf{h}_{t}$ is the output hidden state of an LSTM layer and $\sigma$ is an element-wise logistic function. Finally, the $k$-th element of $\mathbf{y}_{t}$ is used to predict correctness at the next time step given that the next problem ID is $k$. Despite its superior prediction performance over the existing classical methods, DKT has been criticized by other papers (Wilson et al., 2016; Khajah et al., 2016) for its lack of practicality in educational applications. This is because the output hidden state $\mathbf{h}_{t}$ is inherently hard to be interpreted as the knowledge state, and the model does not give insights into the knowledge interaction.

To make a neural network KT model more interpretable, Dynamic Key-Value Memory Networks (DKVMN) (Zhang et al., 2017) have been introduced by extending memory-augmented neural networks (MANNs) (Graves et al., 2014; Weston et al., 2014). Like DKT, DKVMN uses the original data as input. DKVMN accumulates temporal information from student responses into a dynamic matrix, or the value memory, while embedding skills with a static matrix, or the key memory. DKVMN defines the interaction between value vectors and key vectors using attention weights calculated with cosine similarity, and predicts correctness by passing a concatenation of a weighted sum of value vectors and an embedded skill vector to a multilayer perceptron (MLP). Even though the prediction accuracy of DKVMN has proven to be higher than that of DKT, the use of an MLP for the output of the model still makes it hard to explain the knowledge interaction.

3. Our Proposed Model

3.1. Motivation

To generalize the knowledge state while describing the knowledge interaction intuitively, we suggest a model that projects a student’s knowledge and skills into the same vector space of embedding dimensionality $d$. An important constraint is to contain the skill vectors on the $d$-dimensional positive orthant unit sphere, i.e., they have unit-length and positive coordinates. The logit of a probability estimate for correctness is given by the dot product between the current knowledge state vector and the skill vector of the next problem. This is only possible because both knowledge state and skill vectors lie in the same vector space.

Now, we illustrate why skill vectors are set to unit-length and constrained to a positive orthant: the former makes the logit only dependent on the direction of the related skill vector while the latter assures that learning on one skill does not decrease learning on another. For example, in a 2-D vector space, suppose that Ben has knowledge state $\mathbf{KS}_{2}=(1,1)^{T}$ at $t=2$ after two responses while there are three skill vectors, $\mathbf{s}_{1}=(1,0)^{T}$ and $\mathbf{s}_{2}=(0,1)^{T}$ for the skills 1 and 2, and $\mathbf{s}_{3}=(-1,0)^{T}$ for a third “imaginary” skill as shown in Figure 1. At $t=3$, his knowledge state may change to $\mathbf{KS}_{3}=(2,1)^{T}$ as he answers correctly for skill 1. Then the logit with respect to $\mathbf{s}_{1}$ increases from $\mathbf{KS}_{2}\cdot\mathbf{s}_{1}=1$ to $\mathbf{KS}_{3}\cdot\mathbf{s}_{1}=2$ while that with respect to skill 2 remains the same as $\mathbf{KS}_{2}\cdot\mathbf{s}_{2}=\mathbf{KS}_{3}\cdot\mathbf{s}_{2}=1$. However, the logit with respect to skill 3 would decrease from $\mathbf{KS}_{2}\cdot\mathbf{s}_{3}=-1$ to $\mathbf{KS}_{3}\cdot\mathbf{s}_{3}=-2$, which would then decrease the probability estimate for the correctness of skill 3. This is counter-intuitive since the datasets we are dealing with have a set of skills within the same area, e.g., mathematics.

3.2. Objective

Let the skill of a problem and the correctness at time $t$ be $e_{t}\in\{1,\cdots,N\}$ and $c_{t}\in\{0,1\}$, respectively. The correctness $c_{t+1}$ at each time step $t=1,\cdots,T-1$ is viewed as a Bernoulli variable given the history of responses and the skill at time step $t+1$. Then the objective of a KT model is to find out the parameter of the Bernoulli distribution at each time step as follows:

[TABLE]

3.3. Architecture Overview

KQN consists of three components: knowledge encoder, skill encoder, and knowledge state query. The knowledge encoder converts the temporal information from student responses into a knowledge state vector while the skill encoder embeds a skill into a skill vector. The two vectors are then passed to the knowledge state query to provide the prediction for correctness given the current knowledge state and the provided skill. The network architecture of KQN is shown in Figure 2.

3.4. Inputs

The model takes two inputs: a student response at the current time step and a skill at the next time step. Each of the student responses is one-hot encoded and given as input $\mathbf{x}_{t}$ to an RNN layer as the following:

[TABLE]

where $N$ is the number of skills, and $k$ is the skill at time step $t$. Similarly, the skill $k^{\prime}$ at time $t+1$ is one-hot encoded to $\mathbf{e}_{t+1}$, where the $k^{\prime}$-th element is 1 and the other elements are 0’s. In Ben’s case, his response at $t=1$ and the skill for the problem at $t=2$ are encoded to $\mathbf{x}_{1}=(1,0,0,0)^{T}$ and $\mathbf{e}_{2}=(0,1)^{T}$, respectively.

3.5. Knowledge State Query

Let the knowledge state vector $\mathbf{KS}_{t}$ and the embedded skill vector $\mathbf{s}_{t+1}$, both $d$-dimensional, be the two vectors encoded by the knowledge encoder and the skill encoder, respectively. Then the knowledge interaction is defined by the inner product of the two vectors. The logit $y_{t+1}$ and the corresponding prediction probability $p_{t+1}$ are calculated as follows:

[TABLE]

where $\sigma(u)=\frac{1}{1+\exp(-u)}$ is a logistic function and $\cdot$ refers to the inner product. In this way, knowledge interaction is well-defined for the following reasons:

• •

If two skills are independent, their corresponding vectors are orthogonal to each other. Accordingly, an increase or a decrease in the logit with respect to one vector does not affect the logit with respect to the other vector.

• •

If two skills are similar from the probabilistic perspective, then an increase in a logit with respect to one vector would lead to an increase in the logit with respect to the other vector, and vice versa.

Note that from the definition of the knowledge interaction above, it is implied that there can be at most $d$ mutually independent skills. For different values of $d$, whether or not KQN learns the pairwise relationships between skills represented by pairwise distances was tested in experiments and shown in later sections.

As a result, KQN is approximating the parameter of the Bernoulli distribution at each time step as follows:

[TABLE]

3.6. Knowledge State Encoder

Given input $\mathbf{x}_{t}$, the knowledge state encoder produces a knowledge state vector $\mathbf{KS}_{t}$ with the following equations:

[TABLE]

where $\mathbf{W}_{h,KS}\in\mathbb{R}^{d\times H_{\mathit{RNN}}}$, $\mathbf{b}_{h,KS}\in\mathbb{R}^{d}$, $\mathit{RNN}$ is an RNN layer, and $H_{\mathit{RNN}}$ is the state size of $\mathit{RNN}$. In KQN, LSTM (Hochreiter and Schmidhuber, 1997) and Gated Recurrent Units (GRU) (Cho et al., 2014) have been tested as RNN variants. Also, to avoid overfitting, dropout regularization (Srivastava et al., 2014; Zaremba et al., 2014) has been used for the RNN output layer as was used for DKT in a previous work (Xiong et al., 2016).

3.7. Skill Encoder

The skill encoder embeds input $\mathbf{e}_{t+1}$ to $\mathbf{s}_{t+1}$ with an MLP as follows:

[TABLE]

where $\mathbf{W}_{0}\in\mathbb{R}^{H_{\mathit{MLP}}\times N}$, $\mathbf{W}_{1}\in\mathbb{R}^{d\times H_{\mathit{MLP}}}$, $\mathbf{b}_{0}\in\mathbb{R}^{H_{\mathit{MLP}}}$, $\mathbf{b}_{1}\in\mathbb{R}^{d}$, and $\mathit{ReLU}$ is an element-wise ReLU activation with $\mathit{ReLU}(u)=\max(0,u)$ (Nair and Hinton, 2010). Note that $\mathbf{s}_{t+1}$ is now constrained to the $d$-dimensional positive orthant unit sphere, which we will call $\mathbf{U}^{d}$ for the rest of this paper for notational convenience.

3.8. Optimization

At each time step, the cross-entropy error given the probability estimate and the target correctness is calculated, and the error terms for $t=1,\cdots,T-1$ are summed to give the total error.

[TABLE]

The gradients of the total error with respect to the model parameters $\boldsymbol{\theta}_{model}$ have been computed with back-propagation to be used by an optimization method.

4. Probabilistic Skill Similarity

Based on the architecture of KQN, we hereby introduce a novel concept called probabilistic skill similarity to measure the distance between skills from the probabilistic perspective.

4.1. Distance Measures for Skill Vectors

For any two skill vectors $\mathbf{s}_{1},\mathbf{s}_{2}$ learned from KQN, the cosine distance differs from the squared Euclidean distance by only a factor of 2 since they are constrained to $\mathbf{U}^{d}$ as follows:

[TABLE]

4.2. Distances and Odds Ratios

Next, we show how a pairwise distance between two skill vectors is related to the logarithm of their odds ratio. Given a knowledge state vector $\mathbf{KS}\in\mathbb{R}^{d}$ and a skill vector $\mathbf{s}\in\mathbf{U}^{d}$, the probability estimate $p$ for correctness and the corresponding odds $o$ are calculated as follows:

[TABLE]

Then for any two skill vectors $\mathbf{s}_{1},\mathbf{s}_{2}\in\mathbf{U}^{d}$, the logarithm of the odds ratio is characterized by their distance as follows:

[TABLE]

where $\boldsymbol{\Delta}_{1,2}=\frac{\mathbf{s}_{1}-\mathbf{s}_{2}}{||\mathbf{s}_{1}-\mathbf{s}_{2}||}$. Therefore, we say that two skills are probabilistically similar if they are ‘close’ enough based on the distance between their corresponding vectors.

5. Experiments

KQN has been tested for four tasks: correctness prediction, knowledge interaction visualization, skill domain analysis, and the sensitivity analysis of the dimensionality of the vector space. For correctness prediction, the performance of KQN was compared to that of other models on four public datasets: one synthetic and three real-world ones which are available online. Then for a sample student, knowledge interaction was visualized with a heat map to demonstrate the knowledge state query with respect to different skills. Next, the skill domain was explored with clustering based on skill distances. Finally, pairwise distances of the skill vectors in one dimensionality were compared to those in other dimensionalities to conduct the sensitivity analysis of the vector embedding dimensionality.

5.1. Datasets

The following four datasets have been used to evaluate models: ASSISTments 2009-2010, ASSISTments 2015, OLI Engineering Statics 2011, and Synthetic-5. To make a fair comparison of the correctness prediction task, we used the ones provided by the DKVMN source code available online111https://github.com/jennyzhang0215/DKVMN. The statistics of the datasets are shown in Table 2.

5.1.1. ASSISTments 2009-2010

It is a dataset222https://sites.google.com/site/assistmentsdata/home/assistment-2009-2010-data/skill-builder-data-2009-2010 collected by the ASSISTments online tutoring systems (Feng et al., 2009). It was gathered from skill builder problem sets, where students work on the problems to achieve mastery, a certain level of performance, working on similar questions. During the preprocessing, those records without skill names have been discarded. After a problem with duplicate records had been reported by a paper (Xiong et al., 2016), the dataset has since been corrected by the ASSISTments system. Therefore, those results reported by a number of previous papers are not compared in this paper.

5.1.2. ASSISTments 2015

Compared to ASSISTments 2009-2010 which has 110 distinct skill tags, this dataset333https://sites.google.com/site/assistmentsdata/home/2015-assistments-skill-builder-data contains 100 distinct ones with more than twice the number of student responses. Data records with invalid correct values that are not in $\{0,1\}$ have been removed.

5.1.3. OLI Engineering Statics 2011

This dataset444https://pslcdatashop.web.cmu.edu/DatasetInfo?datasetId=507 was gathered from a college level statics course in Fall 2011 (Koedinger et al., 2010). The concatenation of a problem name and a step name has been labeled as a skill tag. Note that the number of skills is much larger than those of other datasets.

5.1.4. Synthetic-5

It is a dataset555https://github.com/chrispiech/DeepKnowledgeTracing originally generated by the authors of the DKT paper (Piech et al., 2015). Each student response was generated using skill difficulty, student proficiency, and the probability of a random guess set to a constant based on IRT (Hulin et al., 1983). The dataset consists of a number of sub-datasets, and those with five concepts from version 0 to version 19 have been used, i.e., a total of 20 sub-datasets from the original dataset were used.

5.2. Setup and Implementation

All the program codes for the implemented KQN and DKT were written in TensorFlow 1.5666https://www.tensorflow.org/versions/r!1.5/. For the data splits of each dataset, we used the same ones00footnotemark: 0 used by DKVMN for a fair comparison of prediction accuracy.

5.2.1. Correctness Prediction

All the sequences of student responses were preserved in their original length without truncation. Each dataset except Synthetic-5 has been split into training, validation, and test sets with 8:2 and 7:3 for training to validation and (training+validation) to test ratios, respectively. For Synthetic-5, the corresponding ratios of 8:2 and 5:5 have been set. Hyperparameters have been grid-searched with holdout validation with early-stopping. Note that no early-stopping was used in the testing phase. The number of epochs was set to 50 and 200 during the validation and testing phases, respectively. During the testing phase, KQN was run for five times, and the mean and standard deviation of the performance metric results have been reported.

The hyperparameters of KQN and their candidate values have been set as follows:

• •

Type of the RNN layer in the knowledge state encoder: LSTM, GRU.

• •

Hidden state size $H_{\mathit{RNN}}$ of the chosen RNN layer: 32, 64, 128.

• •

Hidden state size $H_{\mathit{MLP}}$ of the MLP layer: 32, 64, 128.

• •

Dimensionality $d$ of the vector space in which the knowledge state and skills are embedded: 32, 64, 128.

The retention rate of 0.6 for the RNN dropout and the batch size of 128 were set to default. The Adam optimization method (Kingma and Ba, 2014) was used to minimize the total error $E_{total}$.

Additionally, DKT was run with the skill vectors learned by KQN to evaluate their quality for the correctness prediction task. Specifically, at each time step, input $\mathbf{x}_{t}$ given to DKT was set to the concatenation of two vectors: one-hot encoded correctness $\mathbf{c}_{t}\in\mathbb{R}^{N}$ and the learned skill vector $\mathbf{s}_{t}\in\mathbb{R}^{d}$ corresponding to the original skill $e_{t}$. We denote DKT with such a setup as DKT+KQN. The hyperparameters of DKT+KQN have been searched in the same way as explained previously for KQN with the same dropout rate of RNN and the batch size. Meanwhile, LSTM was used for the RNN layer following past works (Piech et al., 2015; Xiong et al., 2016).

5.2.2. Knowledge Interaction Visualization

Throughout a student’s responses, prediction estimates for correctness with respect to the skills the student solved were calculated with the knowledge state query followed by the logistic function. A sample from the test set of ASSISTments 2009-2010 was used for the task. Then those estimates were visualized with a heat map to evaluate their changes as the student solved the problems.

5.2.3. Skill Domain Analysis

Skill distances have been used for clustering on the four datasets. To decide the linkage and the type of distance measures to use, we first performed flat clustering on Synthetic-5 with the number of clusters fixed to 5, the ground truth number of clusters. Then the quality of clustering or partitioning with respect to the original cluster labels was measured with the Adjusted Rand Index (ARI) (Hubert and Arabie, 1985). It has a maximum value of 1 when the clusters are formed to match the original partitioning perfectly, and a minimum value of 0 when they are randomly partitioned. Since there are 20 sub-datasets for Synthetic-5, 20 ARI scores were averaged. The linkage between clusters and the type of distance measures have been set to hyperparameters as follows:

• •

Cluster linkage: $\{$average, centroid, complete, median, single, ward, weighted$\}$

• •

Type of distance measure: $\{$cosine, Euclidean$\}$

After deciding which linkage and distance measure to use, the number of clusters $n$ has been explored. First, for different values of $n=5,\cdots,14$, the skills of ASSISTments 2009-2010 were clustered based on the distances computed from the skill vectors learned by KQN. Then DKT was used to quantify the quality of those clusters as follows: First, all the original skill IDs were substituted with the assigned cluster labels, where data splits remained the same as those for the correctness prediction task. Then, DKT was run five times. Finally, the average and the standard deviation of the test AUCs of DKT were reported.

5.2.4. Sensitivity Analysis of the Vector Space Dimensionality

Let $d$ be the dimensionality of the vector space in KQN and $d_{opt}$ be the optimal values of $d$ obtained in the correctness prediction task previously. KQN was trained on the four datasets by varying $d$ to $0.5d_{opt}$ and $2d_{opt}$ with the data splits kept the same, and the other hyperparameters set to the optimal values. To analyze the effect of $d$ on the correctness prediction task and the learning of skill vectors, prediction accuracy was reported, and the three distance matrices of the skill vectors for each of $d=d_{opt},0.5d_{opt},2d_{opt}$ were compared.

6. Results and Analysis

6.1. Correctness Prediction