Modeling e-Learners' Cognitive and Metacognitive Strategy in Comparative Question Solving

TL;DR

This paper introduces a novel graph-based data mining approach combining Knowledge Maps and Thinking Maps to automatically detect and visualize learners' cognitive and metacognitive strategies during question-solving tasks in an online environment.

Contribution

It presents a new method that maps cognitive to metacognitive strategies using a graph-based algorithm, enhancing understanding of learners' thinking processes.

Findings

The method successfully identified strategy patterns in student data.

Experimental results support the effectiveness of the proposed approach.

The approach provides a view of learners' cognitive and metacognitive processes.

Abstract

Cognitive and metacognitive strategy had demonstrated a significant role in self-regulated learning (SRL), and an appropriate use of strategies is beneficial to effective learning or question-solving tasks during a human-computer interaction process. This paper proposes a novel method combining Knowledge Map (KM) based data mining technique with Thinking Map (TM) to detect learner's cognitive and metacognitive strategy in the question-solving scenario. In particular, a graph-based mining algorithm is designed to facilitate our proposed method, which can automatically map cognitive strategy to metacognitive strategy with raising abstraction level, and make the cognitive and metacognitive process viewable, which acts like a reverse engineering engine to explain how a learner thinks when solving a question. Additionally, we develop an online learning environment system for participants to learn and record their behaviors. To corroborate the effectiveness of our approach and algorithm, we conduct experiments recruiting 173 postgraduate and undergraduate students, and they were asked to complete a question-solving task, such as "What are similarities and differences between array and pointer?" from "The C Programming Language" course and "What are similarities and differences between packet switching and circuit switching?" from "Computer Network Principle" course. The mined strategies patterns results are encouraging and supported well our proposed method.

Tables

Table 1. TABLE I: Specification of Log Data

Comparision Questions	Number of Subjects	Number of Records
Question I	$104$	$7664$
Question II	$69$	$3208$

Table 2. TABLE II: Use of Metacognition Strategy Patterns

Metacognition Strategy Pattern	Percentage	Sum
Description-Comparison-Description	$31.3\%$
Comparison-Description-Description	$31.0\%$	$93.9\%$
Description-Description-Comparison	$31.6\%$

Equations

$$LAS=<k{u}_1,k{u}_2,\dots ,k{u}_n>$$

$$K{M}_{sub}^{descriptive}=(K{U}_{sub}^{desc},K{E}_{sub}^{desc})$$

$$K{M}_{sub}^{connective}=(K{U}_{sub}^{conn},K{E}_{sub}^{conn})$$

$$CCM=(\frac{N_1}{|K{U}_{sub1}|-k},\frac{N_2}{|K{U}_{sub2}|-k},\dots ,\frac{N_i}{|K{U}_{subi}|-k})$$

$$S_{cog}=<CC{M}_1,CC{M}_2,\dots ,CC{M}_n>$$

$$S_{meta-cog}=(co{g}_1,co{g}_2,\dots ,co{g}_n)$$

$$f(x_{i,j})=\{\begin{array}{ll}1& x_{i,j}=1\\ 0& x_{i,j}=0\\ 0.5& x_{i,j}\in (0,1)\end{array}$$

Full text

I Introduction

In recent years, cognition, metacognition and their strategies have attracted a great deal of research interests from academia and industry [1, 2, 3]. Cognition is about a thinking process, and metacognition is defined as thoughts about thoughts or cognition about cognitions [4]. The lack of e-learners’ cognitive and metacognitive strategy monitoring or without self-monitoring/self-assessment often leads to lower learning efficiency [5, 6]. Hence, it is crucial to detect and model cognitive/metacognitive strategies, in order to make better human-computer interaction, such as providing a means for intelligent tutors to assess learners’ current knowledge and skill levels, or helping students improve learning experience and effectiveness. Traditionally, self-report questionnaires [7, 8] and process tracing methodologies [9, 10] are adopted to analyze the learners’ learning behaviors or cognition. More recently, researchers have been developing data-mining techniques, including hidden Markov models [11], sequential pattern mining approaches [12, 13], coherence analysis [14], clustering [15, 16, 17] and fuzzy decision trees method [18].

However, in most extant research aforementioned modeling cognitive and metacognitive strategy [19, 20], and cognitive activities (goal-setting & planning, knowledge construction, monitoring and help-seeking) [12, 21] were usually modeled in a general cognitive strategy level along with its procedural knowledge (e.g., cognitive activity-based sequence pattern), while the metacognitive strategy sits at the same level with cognitive activities absence of hierarchical model hinders manifestation of their relationships. We agree with the idea, proposed by Kinnebrew et al. [22], that raising the level of abstraction with context summarization from raw log events to a canonical set of distinct actions is a vital first step for effective analysis. Even though the issue with hierarchy has been resolved, there still exist three main obstacles. Firstly, cognitive strategies can be developed in a generic way to serve different domains or specific to a domain, but the current approaches of modeling specific cognitive strategy seldom consider the domain knowledge presentation or construction for question solving when they apply general cognitive strategy. Secondly, metacognitive strategies are in form of complex constructs, not directly observable and problematic to extract. Current research mainly focuses on human expert’s interpretation and manual mapping between cognitive strategies and metacognitive strategies [12, 21, 23, 24]. There is a lack of systematic enabling method that automates mapping of cognitive strategies to these metacognitive strategies by considering knowledge constructions in the context of the selected subject, scope, and sequence (cognition strategy knowledge and domain knowledge itself), when abstracting learner cognitive strategies from a raw log. Thirdly, there lacks a visual language to help portray cognitive and metacognition process and their characteristics, especially for learning approaches.

Aimed at addressing these three problems, an approach to combine Thinking Maps (TM)[25] with Knowledge Maps (KM) [26, 27] is proposed to explore the structure and process of e-learner cognitive strategy by applying TM-based high-level abstraction in the context of KM. Generally, TM is a common visual language for learning communities to describe and visualize the thinking process, and a KM is a directed graph as well as a semantic network composed of nodes and edges. Each node is a knowledge unit, and each edge represents a semantic relation between two knowledge units. More detail on this will be discussed in Section III-C.

Combining TM and KM may throw a light on detecting or recognizing individual cognitive and metacognitive strategy simultaneously, especially in the scenarios of question solving. Inspired by this idea and based on our team’s prior works [19, 26, 28, 29, 30], we introduced the theory of TM to detect and model cognitive and metacognitive strategies with learners’ question solving behaviors acquired from our KM-based e-learning system which includes the proposed mining methods for cognitive and metacognitive strategies. This paper mainly focuses on how learners solve questions by using their cognitive and metacognitive strategies, so we eliminate factors such as ambiguity of natural language, and complexity of Natural Language Processing (NLP) to obtain the effectiveness of the proposed methods and algorithms. In the experiments, simple questions, like “What are similarities and differences between array and pointer?” and “What are similarities and differences between packet switching and circuit switching?” are used to test learners and the system.

Below, we summarize our major contributions in this paper.

Propose a novel method that introduces a theory of thinking maps into the KM-based data mining framework to detect and model cognitive and metacognitive strategies in scenario of question solving. In which, a Knowledge Map is considered as a domain-specific semantic knowledge structure that is not only a kind of knowledge construction used for learning, but also a knowledge base which an e-learner selects the learning scope by using his/her cognitive strategies to solve questions. Thinking Maps is a set of generic cognition strategies with specific knowledge units and it acts as a bridge for mapping cognition strategies to metacognitive strategy that can be automated. 2. 2.

Provide a graph-shape description to different cognitive strategies based on TM (cognitive planning with specific knowledge units) as well as a sequence-style metacognitive strategy (procedural knowledge of individual cognitive strategy) based on mined results. The TM theory provides a visual language to help reveal e-learners’ question-solving behaviors and process of cognitive and metacognitive. Their characteristics are not only exhibited in a general-level but also in a domain-specific-level, simultaneously. The mined results are comprehensive so that they can help other students with learning difficulties to compare and evaluate their cognitive and metacognitive strategies with others. 3. 3.

Coin the corresponding definitions for designed data mining algorithms. We developed an online learning environment for learners and logged their learning behaviors when they carry out assignments task or solving questions. The experimental results are quite encouraging, as they demonstrate the effectiveness of our proposed frameworks and corresponding methods.

The rest of the paper is organized as follows. Section II introduces some of related terminology of cognitive psychology, and then reviews the state-of-the-arts methods of detecting cognitive and metacognitive strategy and their limitations, and then proposes our method. Section III is devoted to explaining the motivation of this paper with the definition of 11 key concepts through a representative example, presenting our algorithm for modeling and detecting cognitive and metacognitive strategy. Section IV describes the experimental setups and summarizes the important results. Section V presents the conclusion, discuss the limitation and future works.

II Background and State of the Art Review

First, we will briefly introduce the basic notions, including cognition, metacognition, cognitive strategy and metacognitive strategy, general cognitive strategy and domain-specific cognitive strategy, as well as their comparison. After that, related works and detecting methods are reviewed in Section II-D.

II-A Cognition and Cognitive Strategy

Cognition is a “mental process of acquiring knowledge and understanding through thought, experience, and the senses” [31]. The thinking processes analyzed from different perspectives within different contexts, notably discussed in the fields of psychology, education and philosophy, etc. The term “cognitive strategies” in its simplest form is the use of the mind (cognition) to solve a question or complete a task. Pressley and Woloshyn [32] hold the view that cognitive strategies can be general or specific. General cognitive strategies can be applied across many different disciplines and situations, such as summarization or setting goals for what to accomplish, whereas specific cognitive strategies tend to be narrow, and they are specific to a task or a domain, such as drawing a picture to help one see how to tackle a physics problem. A specific cognitive strategy should also include the representation or construction of domain knowledge.

II-B Metacognition

In the 1970s, Flavell et al. [4] coined the definition of metacognition as “cognition about cognitive phenomena,” or more simply “thinking about thinking”. Metacognition refers to higher order thinking that involves active control over the cognitive processes involved in learning. Metacognition has two constituent parts: knowledge about cognition and cognitive regulation [33, 34, 35]. Knowledge of cognition includes declarative cognitive knowledge and procedural cognitive knowledge [36, 37].

Procedural knowledge involves awareness and management of cognition, including knowledge about strategies [36]. Cognitive regulation concerned individual ability of planning, monitoring or regulating, and evaluating is a component of metacognition. Especially, the ability of planning means that learner can identify and select appropriate strategies and allocate resources for learning. This feature enables procedural knowledge to interact with strategy knowledge to apply different types of knowledge according to findings [4] (e.g., according to the current outcome, it is better to use strategy A over strategy B to carry out task X). Researchers have found that problem-solving activities require self-regulated learning to be aware of the problem itself and to manage one’s cognitive processes, and they suggested that metacognition can guide goal-directed thinking when people are learning about or solving their problems [1].

II-C Comparison of Cognitive Strategy and Metacognitive Strategy

Cognitive strategies are the basic mental abilities we use to think, study, and learn (e.g., making associations between or comparing different pieces of information), which provide a structure for learning how to solve non-intuitive problems. The strategies help an individual achieve a goal (e.g., solving a math question) by analyzing and evaluating the required information individually. In contrast, metacognitive strategies are used to ensure that an overarching learning goal can be achieved. Examples [38] of metacognitive activities include planning of how to approach a learning task, using appropriate skills and strategies to solve a problem, utilizing self-assessing and self-correcting in response to the self-regulation, evaluating progress toward the completion of a task, and becoming aware of distracting stimuli etc. The required procedural knowledge of cognition to fulfill these abilities, however, is more complex than a sequence-shape structure.

One of cognitive strategies, self-monitoring, can be used to monitor metacognitive strategies. For example, one of the major functions of self-regulation is to check themselves if they have solved questions. However, metacognitive strategies with complex constructs are not easy to extract due to their implicit semantics. This leads to that the existing research cannot automate the process of mapping between cognitive strategies and metacognitive strategies [24, 21], but manually.

II-D Modeling and Detecting Methods Review

The traditional approaches to measuring learner’s cognitive and metacognitive strategies are based on questionnaires. For example, Zimmerman and Martinez-Pons [39] conducted pilot interviews with high school students. They were asked to indicate in each topic which approaches they used to participate, to study, and to complete their assignments. Though the questionnaire approach that achieved 93% accuracy using self-regulated learning (SRL) strategies, it failed to capture the dynamic and adaptive nature of SRL, as it did not consider the learning processes of students’ knowledge-building, and problem-solving. Increasingly, researchers have focused more attention on analyzing the data collected from tracing students’ learning process in a computer-based learning environment. For example, Perry and Winne [10] categorized the activity traces into four statistical ways, which are frequency of study events, patterns of study activity, time and sequence of events, and content analysis of students’ notes and summaries. Results suggest that analytic on trace data give better comprehension about how learners select, monitor, assemble, rehearse, and translate information. This provides useful raw materials for mapping SRL to learning effects.

In order to better understand learning process, data mining techniques have been adopted to measure learning or question-solving behaviors. Perera et al. [15] applied k-means clustering to find groups of similar teams and similar individual members, and then used sequential pattern mining algorithms to extract sequences of frequent events. Specially, Kinnebrew and Biswas have been researching metacognitive and SRL strategy in the past decade, which includes a series of methods to analyze learning behaviors. In 2011, they incorporated HMMs and exploratory data mining methodology for assessing and comparing students’ learning behaviors from traces [11]. They employed a combination of sequence mining techniques to identify differential frequent patterns between groups of students in [22], analyzed the evolution of students’ frequent behavior patterns over time in [13], used coherence analysis to characterize SRL behaviors in [14], developed a framework that combines model-driven strategy detection with data-driven pattern discovery for analyzing students’ learning activity data in [40], etc. Segedy et al. [21] proposed an integrated cognitive and metacognitive model for effective, self-regulated student learning in the Betty’s Brain environment, while manually designing the mapping between the metacognitive and cognitive activities in their project that includes four aspects: goal-setting and planning, knowledge construction, monitoring and help-seeking. Particularly, in order to evaluate metacognition while individuals are learning a new task, Kim et al. [1] proposed a fuzzy linear regression model to analyze the relationship between retrospective confidence judgments (RCJ) and situation awareness (SA).

However, most of the existing researches seldom consider domain-specific knowledge constructions, they lack an automatic mapping mechanism between metacognitive strategies and activities, or fail to make the cognitive and metacognitive process visualized. These challenges have not been addressed. For example, the research from Kinnebrew et al. have a visual causal map that represents the relevant science phenomena as a set of entities connected by directed links that represent causal relations, called Betty’s Brain [41]. It, however, was unable to overcome the other two challenges [42, 43].

In the constructivist’s viewpoint, an integrated approach to the construction of procedural knowledge that should include discovery and acquisition of appropriate domain knowledge as well as subject choice, content scope selection, and sequence of learning. Therefore, the challenges of forming metacognition are not only to elicit procedural knowledge, but also to determine the domain scope and acquire appropriate knowledge.

With the development of knowledge engineering and educational psychology research, some recent outcomes, such as Thinking Maps (TM) and Knowledge Maps (KM), provide some basis of recognizing cognitive and metacognitive strategies. A knowledge map consists of the knowledge units of a given domain and the learning dependencies among them, which is a directed graph [26] (a kind of domain knowledge structure). A KM is known to be correlated with a range of human endeavors [27] and one emerging discipline that integrates different disciplines (e.g. economic, psychology, engineering etc.). Liu et al. had indicated that understanding the topological properties of knowledge maps may help gain better insights into human cognition structure and its mechanism [26]. Only few research efforts utilizing KM to explore human cognition and metacognition process and the corresponding strategies, have been reported until now. Meanwhile, Thinking Maps claim to be a method to exploit higher order thinking skills and acts as a visual language to enhance learning experiences [25]. Therefore, the above research inspired us to combine KM with TM to explore the structure and process of e-learner’s cognitive strategy by applying TM-based high-level abstraction in the context of KM. This could reveal how students use their cognitive and metacognitive strategies during question solving, especially in a kind of scenario of problem-based learning [44]. This paper provides an effective framework for detecting cognitive and metacognitive strategy in the context of comparative question solving.

III Framework and Algorithm

III-A Motivation

Our proposed framework is shown in Fig. 1, which tries to explain how e-learners think, when solving questions, and illustrates our motivation inspired by a three-layer reverse engineering. The three-layer diagram consists of “Learning Activity Sequences” layer, “Cognitive Procedure” layer and “Metacognitive Strategy” layer. The hierarchical relationship from “Learning Activity Sequences” to “Metacognitive Strategy” abstractly describes a data-mining-based analysis process that maps specific behaviors and sequences of learning into cognitive elements and procedure by applying a bottom-up method. For example, an e-learner’s knowledge units of learning activity sequence is generated when solving the comparative question “What are similarities and differences between array and pointer?”, Array Definition, Array Type,…, 2D Array Initialization, etc., shown in Fig. 1, are recorded into a learning log, then a data-mining-based method on KMs and TMs analyzes these learning activity sequences and automatically maps them into different cognitive strategies, such as “Description of Array” and “Comparison of Array and Pointer”, sequentially. This forms an instance of cognitive procedure for solving a comparative question, i.e. an instance of cognitive planning with specific knowledge units. After that, another method from the Thinking Map can be applied to map the cognitive procedure into the procedural cognitive knowledge comprised of “Description-Comparison-Description” without specific knowledge units, which is a metacognitive strategy the learner used before or stored in individual mind. Therefore, we can acquire the e-learner’s cognitive and metacognitive strategy at different layers.

III-B Notation and Definition

This section introduces notations and definitions used throughout this paper.

Definition 1** (Knowledge Unit)**

A knowledge unit (denoted as $ku$) is the smallest integral learning object, such as a definition, a theorem or an algorithm [28]. It is a node or entity in a graph, which usually composed of both name and content, and a core term of a $ku$ is its name. Note that one knowledge unit may only have one core term.

For example, a $ku$ Array Definition in “The C Programming Language” course knowledge map can be described as “Name: Array Definition; Content: An array is a group of variables of a type occupying a contiguous region of memory [45]”, and the core term of the knowledge unit is “array definition”.

Definition 2** (Semantic Relation Between Two Kus)**

A semantic relation (denoted as $r$) is a directed edge in graph/map, which indicates a relationship from one $ku$ to another $ku$.

The type of semantic relation is diverse, commonly occurring are: “a part of”, “an attribute”, “a definition”, “a kind of”, “a type of”, “an association”, “similar to”, “an initial cause”, and “a result”, etc. For example, as shown in Fig. 2, a circle can be divided into two different types: inscribed circles and circumscribed circles. Therefore, the semantic relation between Circle $ku$ and Inscribed Circle $ku$ (or Circumscribed Circle $ku$) can be called “a kind of”.

Definition 3** (Knowledge Map)**

A knowledge map is a knowledge structure of a course or discipline organized and managed its knowledge unit in a graph-based manner [26] (denoted as KM). A KM is a directed graph composed of entities (nodes) and relations (edges). Besides, it can be viewed as a semantic network, in which each node is a knowledge unit and edge is represented as a triple of the form (head $ku$, semantic relation, tail $ku$), denoted as $ke$. Formally, we define a KM: $KM=(KU,KE)$, in which, $ku\in KU$ and $ke\in KE$.

Fig. 3 depicts a partial KM of “The C Programming Language” course, in which, each edge has a semantic relation, for example, the edge between Function and Body will be represented as (Function, “a part”, Body).

Definition 4** (Learning Activity Sequence)**

A learning activity sequence obtained from a learning log reflects a track of a student’s learning process during a period, which is an output of the first layer in Fig. 1. A $LAS$ containing $n$ $kus$ can be represented as follow:

[TABLE]

For example, an instance of a specific student’s $LAS$ may be $<$Array Definition, Array Type, 2D Array, 2D Array Initialization, Array Pointer, Array Pointer Structure, Pointer Array$>$ to answer “What are similarities and differences between array and pointer?” question, shown in Fig. 4.

Definition 5** (Thinking Map)**

Thinking Map (denoted as TM) is a common visual language for learning communities to describe and visualize the thinking process [25].

Fig. 5 shows eight kinds of graph methods: Brace Map, Bubble Map, Circle Map, Tree Map, Bridge Map, Multi-Flow Map, Double Bubble Map and Flow Map. Each map represents a basic thinking process when thinking about questions, and the corresponding eight thinking processes are Whole/Part, Describing Qualities, Context/Frame of Reference, Classification, Analogies, Cause and Effect, Compare and Contrast and Sequencing.

A KM describes the knowledge units and their semantic relationships within a specific domain. While each thinking map means nothing if the corresponding process does not include the specific knowledge, especially when a person is thinking about how to answer the domain specific questions. Hence, the methodology of combining KMs with TMs can model a person’s thinking process in solving questions, when they ponder about them.

To deal with this, there are three things that need to be considered: questions, mapping from questions to KM, and mapping from question-oriented KM to TM.

In this paper, we concentrate on how e-learners solve problems by using their cognitive and metacognitive strategies. To gain better understanding this research and its result, we adopt less complex questions to test e-learners. For mapping from questions to KM, the core terms will help to locate the corresponding knowledge units, according to Definition 6. For mapping from question-oriented KM to TM, we utilize a mapping relationship between KM and TM. Particularly, in Fig. 6, we illustrated the eight kinds of Thinking Maps with specific questions in the context of the KM of “Computer Network Principle” course.

Take the Bubble Map in Fig. 6 as an example. Given a question, such as “How do you know about protocol?”, then a learner may invoke the knowledge related to “protocol” and construct the bubble-like map. In this scenario, firstly, abstract the core term “protocol” from the question; secondly, extract the knowledge units corresponding to the “protocol” course located in KM; thirdly, mine or find a specific knowledge submap in the KM by using the relation mapping between KM and TM. In this process, we call knowledge unit Protocol a core knowledge unit, and the specific knowledge submap mined in KM is deem to a thinking map with core knowledge units.

Based on the above, we coin some important definitions as followings.

Definition 6** (Core knowledge Unit)**

A core knowledge unit (denoted as $cku$) is a centric phrase of a question sentence, which builds a bridge between a question and knowledge units.

For example, given a specific question, “What is the definition of array?”, it can be inferred that “array” is the core item and “definition” denotes an attribute of “array”, so that, we can call Array is a core knowledge unit, and Array Definition is a subordinate $ku$ of Array. Similarly, the core knowledge units of this question “What are similarities and differences between array and pointer?” are Array and Pointer. The $ckus$ are all colored in yellow this paper.

Definition 7** (Thinking Maps with $cku$)**

A Thinking Map with $cku$ is a special knowledge submap, which can be found in a domain-specific KM by the presented algorithm, in Section III-C.

For instance, we can follow the “a type of” or “a kind of” semantic relation to construct a Tree Map with Protocol $cku$ submap, and follow the “an association” semantic relation to build a Circle Map with Protocol $cku$ submap, depicted in Fig. 6. Consequently, more than one submap is built with the Protocol $cku$. Moreover, We provide a description of TM with $cku$ submap searched algorithm discussed in Algorithm 2.

Particularly, we introduce two specific knowledge submaps: Descriptive Knowledge Submap and Connective Knowledge Submap, which are the basic elements of Bubble Map and Double Bubble Map with $cku$, the detailed description of these submaps are as follow.

A Descriptive Knowledge Submap can be defined by:

[TABLE]

Where $KU^{desc}_{sub}$ is a set of $kus$, which are first-order neighbor vertices of a $cku$ colored in blue, and $KE^{desc}_{sub}$ is a set of semantic relations between $cku$ colored in yellow and other $ku$. That is, it satisfies $\forall ku\in KU^{desc}_{sub},(cku,r,ku)\in KE^{desc}_{sub}$. For example, we searched two submaps from “The C Programming Language” course KM, which are descriptive knowledge submap of Array $KM^{desc(array)}_{sub}$ and descriptive knowledge submap of Pointer $KM^{desc(pointer)}_{sub}$, illustrated within the light yellow area of Fig. 7a and 7c, and the overrange $kus$ are colored in white.

Similarly, a Connective Knowledge Submap can be defined by:

[TABLE]

A connective knowledge submap usually contains two core knowledge units mapped by core terms, such as Array and Pointer units colored in yellow, as shown in the connective knowledge submap of Array and Pointer $KM_{sub}^{conn(array,pointer)}$, Fig. 7b. In which, the connective $kus$ of two $ckus$ are Array Pointer and Pointer Array.

To integrate descriptive submap and connective submap, i.e., merging the Array descriptive knowledge submap, the Pointer descriptive knowledge submap and the connective knowledge submap of the two $kus$, we can obtain a Double Bubble Map with Array and Pointer, exhibited in Fig. 7d, which illustrates the similarities and differences between array and pointer.

Next, we will proposed Cognition Control Measure, Cognition Control Sequence, Cognitive Procedure and Metacognitive Strategy Sequence, to analysis the learning activity sequence and measure the individual question-solving process.

Definition 8** (Cognition Control Measure)**

Cognition control measure (denoted as $CCM$) is a coverage rate of knowledge submaps, which provides a quantitative calculation of cognition. The definition is as below:

[TABLE]

Where, $N_{i}$ denotes the number of $ku$ which belong to the responding $KM_{subi}$ at a learning moment; $|KU_{subi}|$ denotes the numbers of units in $KM_{sub}$; $k\in\{1,2\}$, denotes the number of core knowledge units in its submap; $i$ denotes the number of searched knowledge submaps, in addition, the type and order of these submaps can be extracted from learning activity sequence.

Take the $LAS$ in Fig. 4 as an example, assuming that the corresponding knowledge submaps are searched and represented in Fig. 7a, 7b and 7c. As can be seen, the number of $kus$ in $KM^{desc(array)}_{sub}$ is 6 (not including Array $cku$), and there are 3 $kus$ in current learning activity sequence, so the knowledge unit coverage rate of $KM^{desc(array)}_{sub}$ is: $RatioSub_{desc(array)}=\frac{1}{2}$ . Similarly, the number of $kus$ in $KM^{conn(array,pointer)}_{sub}$ is 5 (not including Array $cku$ and Pointer $cku$), and there are 3 $kus$ in current learning activity sequence, so the knowledge unit coverage rate of $KM^{conn(array,pointer)}_{sub}$ is: $RatioSub_{conn(array,pointer)}=\frac{3}{5}$. However, there are no learning activities in the Knowledge Submap describing Pointer, so the knowledge unit coverage rate of Pointer $KM^{desc(pointer)}_{sub}$ is: $RatioSub_{desc(array)}=0$ . Hence, the $CCM$ of this learning sequences in Fig. 4 is $(\frac{1}{2},\frac{3}{5},0)$.

Definition 9** (Cognition Control Sequence)**

Cognition Control Sequence (denoted as $S_{cog}$ ) reflects the dynamic process of completing a certain task, hence, a $S_{cog}$ consists of $CCMs$ of different moments. The representation of $S_{cog}$ is given below:

[TABLE]

Definition 10** (Cognitive Procedure)**

Cognitive Procedure also called cognitive planning with specific knowledge units. It reflects the dynamic changes of the learners’ cognitive strategies, which is an output of the second layer in Fig. 1.

Based on TM with $cku$ and the $S_{cog}$, we can abstracted that “Description of Array” and “Description of Pointer” are specific cognitive strategies depicted by Bubble Maps with Array and Pointer respectively, and “Comparison of Array and Pointer” cognitive strategy is depicted by a Double Bubble Map with the two core knowledge units.

Definition 11** (Metacognitive Strategy Sequence)**

A metacognitive strategy sequence is represented by:

[TABLE]

Where, $cog$ is the selected cognitive strategy, and the order of sequence is combinations of cognitive strategies. $S_{meta-cog}$ is the output of the third layer in our framework. From which, generally under the scenario of solving a complex question, a combination of different metacognitive strategies can be mined through further abstraction. Therefore, the metacognitive strategy of solving “What are similarities and differences between array and pointer?” may be represented as (Description, Comparison, Description).

III-C The Proposed Pattern Mining Algorithm

This section presents our cognitive strategy and metacognition strategy patterns mining algorithm under the scenario of question solving. Algorithm 1 describes a main structure of our mining algorithm, which follows the reverse engineering engine idea to raise the abstraction level from learning activities to metacognitive strategy pattern. Algorithm 2 finds the candidate submaps through a heuristic search method, which is guided by Thinking Map. Given specific core knowledge units, the corresponding candidate submaps with $cku$ will be found. Algorithm 3 transforms a three-dimensional vector into a one-dimensional sequence, to reduce the time and space complexity. Algorithm 4 encodes the knowledge unit coverage rate and Algorithm 5 decodes the Cognition Control Sequence.

In Algorithm 1, Step 1 finds the domain Knowledge Map based $courseId$ that points to a specific knowledge map; Step 2 to 5 aim to search the core knowledge unit in KM mapped by $CI$ representing a core item set; Step 6 searches the candidate submaps by calling $SubmapSearch$ algorithm presented in Algorithm 2; Step 7 to 11 generates the cognition control sequence by calculating knowledge unit coverage rate in $KM_{sub}$, and get rid of the irrelevant submaps, so that we obtain the cognitive strategy sequence in step 12; Step 13 to 15 encode the dimensional cognition control sequence described specifically in Algorithm 3 $EncodeCogSeq$; Step 16 mines frequent patterns from the encoded cognition control sequence by the sequential pattern mining algorithm GSP (Generalized Sequential Patterns) [46]. From the definition of knowledge unit coverage rate, the cognition control sequence of comparison question is a three-dimensional vector, we transform the cognition control sequence into a one-dimensional sequence defined by the algorithm in Algorithm 4 $Seg$; Step 17 decodes the frequent patterns to get the metacognition strategy pattern in Algorithm 5 $DecodeCogPattern$. Therefore, we obtain the metacognition strategy pattern in the last Step.

Algorithm 2 detailedly describes the process of finding relevant submaps with specific $cku$ from the eight Thinking Maps defined in Definition 5.

IV Experiments

The following experiments were carried out to validate the proposed method in the context of simple questions, where “simple” means the core items of a question are explicit and without any ambiguity. This paper selects two simple and typical comparison questions from two courses. Question I is “What are similarities and differences between array and pointer?”. Question II is “What are similarities and differences between packet switching and circuit switching?”.

IV-A Data and Online Learning System

A developed online learning environment system based on KM is shown in Fig. 8, where Fig. 8a illustrates “The C Programming Language” course, and Fig. 8b depicts “Computer Network Principle” course. The left-hand side of “The C Programming Language” interface presents a partial KM of the course, while the right-hand side provides the specific explanation of a $ku$ in KM. Likewise, the left-hand side of “Computer Network Principle” interface shows related teaching resources (e.g. PowerPoint), while the middle block is the explanation of a $ku$, and the right-hand side is a partial KM. Mappings between teaching resources and knowledge units are associated at knowledge unit level, i.e., by clicking a $ku$, a partial KM related to $ku$ will display on the right-hand side. Moreover, the middle part will give the $ku^{\prime}s$ explanation regarding its source and specific meaning in order to help learners better master the knowledge unit. The learning behaviors of learners can be accurately acquired by a log collection tool running in backend. A learning log contains $id$, $user\_id$, $user\_name$, $question\_id$, $action\_type$, $object\_id$, $action\_object$ and $timestamp$.

The log data of these two comparison questions was collected during two semesters from 173 participants from the department of computer science and technology of Xi’an Jiaotong University, and some students from Xi’an University of Posts and Telecommunications. After preprocessing data, such as removing records that are unrelated to the comparison question learning, including log in, exit, submit and post, we obtained 10,872 effective records. Table I shows learning records of Question I and Question II, respectively.

IV-B Evaluation

After conducting the experiments on 173 subjects, three metacognitive strategy patterns are mined from 10,872 learning records, which are “Description-Comparison-Description”, “Comparison-Description-Description” and “Description-Description-Comparison”. Additionally, we found that the most frequently occurred submaps are Bubble-Map-like submaps and Double-Bubble-Map-like submaps from participants’ learning activity sequences in the task of solving Question I or II.

Three representative cases using cognitive and metacognitive strategy to solve the two questions are visualized in Fig. 9a, 9b and 9c, respectively. Each figure is comprised of two parts, the top part is a chart consisting of three curves, and the bottom part is a graph-based structure. The horizontal axis of each curve denotes learning event sequence, and the vertical axis denotes knowledge submap coverage rate. Specially, the blue diamond curve represents the change of the descriptive knowledge submap coverage rate of the first $ku$ (e.g., Array or Packet Switching), $DescRate_{1}$, and the $kus$ of this submap will be tinted with the same dye as soon as being visited; the green x-mark curve represents the change of the descriptive knowledge submap coverage rate of the second $ku$ (e.g., Pointer or Circuit Switching), $DescRate_{2}$, and the $kus$ of this submap will be tinted in green as well once being visited; the brown square curve shows the change of the connective knowledge submap coverage rate of two $kus$, $CompRate$, similarily, the visited $kus$ of this submap will be colored in brown. What’s more, there is a coverage rate threshold line with a value of 0.6, colored in red, which means a valid cognitive strategy is recognized once the coverage rate beyond the threshold.

As shown in Fig. 9a, within the interval [1,12], the descriptive knowledge submap coverage rate of the first $ku$, $DescRate_{1}$, changes from 0 to 1, which means the learner finish the visit of all knowledge units describing the first $ku$ during this phase. We can see in [3,5] interval, the knowledge coverage rate stays the same value, which can be ascribed to the subnodes learning ($ku_{4}$ and $ku_{5}$ are the subnodes of $ku_{3}$). However, the descriptive knowledge submap coverage rate of the second $ku$, $DescRate_{2}$, and the connective knowledge submap coverage rate, $CompRate$, remain [math]. This indicates that learners do not visit the $kus$ within the descriptive knowledge submap of the second $ku$ or the connective knowledge submap of two $kus$. Meanwhile, the corresponding operations on these submaps are illustrated in the bottom part of this figure. Seen from the Double Bubble Map that the $kus$ of descriptive knowledge submap of the first $ku$ are visited first and colored in blue, and other $kus$ are not. It is obvious that this learner select learning activities well and the scope of cognitive strategy called description of first $ku$. Hence, this phase is simply deemed “The First Description Stage”.

Within the interval [13,19], $DescRate_{1}$ and $DescRate_{2}$ keep unchanged, while $CompRate$ changes from 0 to 0.8, which implies learners merely visit the connective knowledge submap of two $kus$. Synchronously, the $kus$ of connective knowledge submap of the first $ku$ and the second $ku$ are visited and colored in red. This phase concentrates on comparison of two $kus$, so it can be called “Comparison Stage”.

Within the interval [20,24], $DescRate_{1}$ and $CompRate$ remain invariable, while $DescRate_{2}$ changes from 0 to 0.8, concurrently, the $kus$ of descriptive of the second $ku$ are visited and colored in green.Likewise, this phase is “The Second Description Stage”. Consequently, the question-solving process is accomplished.

The above three stages in Fig. 9a demonstrate that learners control well in the selection of learning activities and the scope of cognitive strategy, meanwhile, the cognition control sequence is $<$descriptive of first $ku$, connective of two $kus$, descriptive of second ku$$>. Furthermore, this kind of metacognition strategy pattern is named “Description-Comparison-Description”. Similarly, from Fig. 9b and 9c, we can also conclude the second and third kind of metacognition strategy pattern can be called “Comparison-Description-Description” and “Description-Description-Comparison”, respectively.

The experimental results of the three metacognition strategy patterns are shown in Table II. We can see that the percentage of students who adopted the three metacognition strategy patterns are all close to 31%, meanwhile, the total sum of these being 93.9%, which means the majority of students tend to use metacognition strategies to solve the comparative questions and successfully. Learners who adopt “Description-Comparison-Description” pattern prefer a sequential manner, and solve the questions step-by-step, however, learners who use the other two patterns may be in a saltaory thinking, and solve questions in general. Therefore, the mining patterns are interpretable and useful in the context of comparative question solving. At the same time, with regard to the experiment results, the $CCM$ of each description/comparison knowledge submap that students visited is 100%. For knowing the reason of this, we revisited the students, and they said that they tried their best to find all knowledge units possibly related to the questions during the experiment. So, we believe that this is the reason to have a high value to each $CCM$.

V Conclusion and Future Work

This study focuses on how learners solve questions by adopting their cognitive and metacognitive strategies, which follows an idea of reverse engineering. We propose a novel method that combines Thinking Maps with Knowledge Maps to detect and model the cognitive and metacognitive strategies from the learning logs gathered from a computer-based e-learning system. From the experimental results, the idea of raising the level of abstraction is quite important, which verifies that cognitive strategies can be described with a domain specific knowledge map, and metacognition strategies which are complex constructs that are not directly observable, can be obtained from a cognition control sequence that consist of cognitive strategies. This shows that mapping between cognitive strategies and procedural cognitive knowledge (metacognition) can be automated to provide their meaningful interactions and supports. Meanwhile, the cognitive strategies and three metacognitive strategy patterns can be represented in a graph structure. Consequently, this graph data-driven method is proved to be effective, as it provides a way to visualize the thinking process of solving questions.

It should be noted that this research has some limitations. Firstly, we do not consider the differences in student’s background, knowledge and their general cognitive ability. Many researches indicated that the difference has played an important role in the use of cognitive strategies and metacognition strategies. Secondly, we only discuss use of cognitive and metacognitive strategy within the scope of comparative question-solving in this paper. Thirdly, we conduct experiments on simple questions, not considering the ambiguity of questions and reduce its complexity. In future work, we will expand not only the number of participants including various topics for comparison, but also the question types. Additional, we will take natural language related contents into account, to improve our methods and algorithms by natural language processing techniques.

Acknowledgments

This work was supported by the National Key Research and Development Program of China under Grant No. 2016YFB1000903; the National Science Foundation of China under Grant No. 61877048; Innovative Research Group of the National Natural Science Foundation of China under Grant No. 61721002; Innovation Research Team of Ministry of Education (IRT_17R86); project of China Knowledge Centre for Engineering Science and Technology.