FAST$^2$: an Intelligent Assistant for Finding Relevant Papers

TL;DR

FAST$^2$ is an innovative AI-powered tool that significantly accelerates and improves the accuracy of literature reviews in software engineering by guiding initial paper selection, estimating remaining relevant papers, and self-correcting classifications.

Contribution

The paper introduces FAST$^2$, a novel tool with three key innovations: external domain knowledge application, an estimator for remaining relevant papers, and a self-correcting classification algorithm.

Findings

FAST$^2$ speeds up literature review process.

It compensates for human errors in paper classification.

FAST$^2$ is robust across multiple large reviews.

Abstract

Literature reviews are essential for any researcher trying to keep up to date with the burgeoning software engineering literature. FAST$^2$ is a novel tool for reducing the effort required for conducting literature reviews by assisting the researchers to find the next promising paper to read (among a set of unread papers). This paper describes FAST$^2$ and tests it on four large software engineering literature reviews conducted by Wahono (2015), Hall (2012), Radjenovi\'c (2013) and Kitchenham (2017). We find that FAST$^2$ is a faster and robust tool to assist researcher finding relevant SE papers which can compensate for the errors made by humans during the review process. The effectiveness of FAST$^2$ can be attributed to three key innovations: (1) a novel way of applying external domain knowledge (a simple two or three keyword search) to guide the initial selection of papers---which helps to find relevant research papers faster with less variances; (2) an estimator of the number of remaining relevant papers yet to be found---which in practical settings can be used to decide if the reviewing process needs to be terminated; (3) a novel self-correcting classification algorithm---automatically corrects itself, in cases where the researcher wrongly classifies a paper.

Figures

Tables

Table 1. Table 1: Statistics from the literature review case studies in this paper

Dataset	Topic	Original Title	#candidate papers $\left|E\right|$	#relevant papers $\left|R\right|$	Prevalence $\left(\left|R\right|/\left|E\right|\right)\%$
Wahono	Defect prediction	A systematic literature review of software defect prediction: research trends, datasets, methods and frameworks	7002	62	0.9%
Hall	Defect prediction	A systematic review of theory use in studies investigating the motivations of software engineers	8911	104	1.2%
Radjenović	Defect prediction metrics	Software fault prediction metrics: A systematic literature review	6000	48	0.8%
Kitchenham	Literature review	A systematic review of systematic review process research in SE	1704	45	2.6%

Table 2. Table 2: Problem description

$E$:	the set of all candidate papers
	(returned from search).
$R\subset E$:	set of ground truth relevant papers.
$I=E\setminus R$:	set of ground truth non-relevant papers
$L\subset E$:	set of labeled/reviewed papers,
	each review reveals whether a paper
	is included or not.
$\neg L=E\setminus L$:	set of unlabeled/unreviewed papers.
$L_R=L\cap R$:	identified relevant (included) papers.
$L_I=L\cap I$:	identified non-relevant (excluded) papers.

Table 3. Table 3: Testing tactics for “how to start”

			X95		WSS@95
Dataset	Rank	Treatment	median	IQR	median	IQR
Wahono	1	Rank-BM25	630	48	0.86	0.01
	2	FASTREAD	685	225	0.85	0.03
	2	Auto-Syn	705	60	0.85	0.01
	3	Auto-BM25	785	78	0.84	0.01
Hall	1	Rank-BM25	290	10	0.92	0.00
	2	Auto-Syn	320	30	0.91	0.00
	3	Auto-BM25	345	30	0.91	0.00
	3	FASTREAD	345	125	0.91	0.01
Radjenović	1	Auto-Syn	515	122	0.86	0.02
	2	Rank-BM25	615	85	0.85	0.01
	3	FASTREAD	700	208	0.83	0.03
	4	Auto-BM25	800	135	0.82	0.02
Kitchenham	1	Auto-BM25	510	30	0.65	0.02
	1	Auto-Syn	520	40	0.64	0.02
	1	Rank-BM25	525	60	0.64	0.03
	2	FASTREAD	630	130	0.58	0.07

Table 4. Table 4: Comparison of different stopping rules

Dataset	Stopping rule	Final Recall $\left|L_R\right|/\left|R\right|$	#papers reviewed $\left|L\right|$
Wahono	SEMI (90)	94(0)	750(48)
	SEMI (95)	95(0)	1180(67)
	Ros’17	76(8)	370(108)
	Cormack’16	94(11)	770(288)
Hall	SEMI (90)	95(2)	315(20)
	SEMI (95)	98(0)	500(30)
	Ros’17	97(0)	385(48)
	Cormack’16	97(1)	390(38)
Radjenović	SEMI (90)	92(0)	540(58)
	SEMI (95)	94(0)	760(37)
	Ros’17	83(4)	320(75)
	Cormack’16	88(6)	420(100)
Kitchenham	SEMI (90)	89(2)	400(20)
	SEMI (95)	91(2)	510(20)
	Ros’17	89(0)	420(50)
	Cormack’16	96(2)	660(85)

Table 5. Table 5: Comparison of different error correction methods.

		Recall		b/a		Precision		d/c		Effort		f/e
		a=100%	b=70%			c= 100%	d=70%			e=100%	f=70%
None	Wahono	95(0)	69(8)	73(8)		100(0)	90(4)	90(4)		1,180(67)	1,640(335)	139(34)
	Hall	98(0)	70(4)	71(4)		100(0)	95(3)	95(3)		500(30)	1,480(465)	296(100)
	Radjenović	94(0)	66(6)	70(7)		100(0)	90(3)	90(3)		760(37)	1,405(467)	185(66)
	Kitchenham	91(2)	69(4)	76(7)		100(0)	86(6)	86(6)		510(20)	660(140)	129(31)
				73				90				187
Kuhrmann’17	Wahono	95(0)	75(7)	79(7)		100(0)	100(0)	100(0)		2,380(140)	3,193(471)	134(26)
	Hall	98(0)	77(5)	79(5)		100(0)	100(0)	100(0)		980(75)	1,603(525)	164(61)
	Radjenović	94(0)	76(8)	81(9)		100(0)	100(0)	100(0)		1,520(80)	2,223(788)	146(57)
	Kitchenham	91(0)	73(4)	80(5)		100(0)	100(0)	100(0)		1,020(20)	1,208(175)	118(20)
				80				100				140
Cormack’17	Wahono	94(7)	65(12)	69(20)		100(0)	99(4)	99(4)		999(238)	712(360)	71(60)
	Hall	97(1)	84(4)	87(5)		100(0)	99(2)	99(2)		552(58)	660(137)	120(35)
	Radjenović	86(4)	75(10)	87(16)		100(0)	98(3)	98(3)		544(114)	713(275)	131(74)
	Kitchenham	96(4)	87(14)	90(19)		100(0)	85(11)	85(11)		1041(168)	1910(1004)	183(113)
				83				95				126
Disagree	Wahono	95(0)	88(3)	93(3)		100(0)	90(5)	90(5)		2,051(63)	2,531(406)	123(23)
	Hall	98(0)	90(3)	92(3)		100(0)	98(3)	98(3)		755(51)	897(206)	119(34)
	Radjenović	94(0)	88(6)	94(6)		100(0)	91(3)	91(3)		1,390(89)	1,577(336)	113(31)
	Kitchenham	91(2)	81(7)	89(10)		100(0)	85(7)	85(7)		940(30)	975(144)	104(19)
				92				91				115

Equations

$$BM25(i,Q)$$

$$=$$

$$IDF(q_j)=log\frac{|E|-n(q_j)+0.5}{n(q_j)+0.5}$$

Full text

††journal: Expert Systems with Applications

1 Introduction

This article presents and assesses an automatic assistant for researchers seeking research papers relevant to their particular topic. The goal of this assistant, called FAST2, is to reduce the effort required for such a search thereby (a) enabling researchers to find more relevant papers faster; thus (b) allowing the contribution of more researchers to be wildly recognized. Many such assistants have been previously proposed [8, 54, 34, 57], but those prior works did not fully address three core problems solved by FAST2 (see our research questions, listed on the next page).

Why is it important to reduce the effort associated with literature reviews? Given the prevalence of tools like SCOPUS, Google Scholar, ACM Portal, IEEE Xplorer, Science Direct, etc., it is a relatively simple task to find a few relevant papers for any particular research query. However, what if the goal is not to find a few papers, but instead to find most of the relevant papers? Such broad searches are very commonly conducted by:

• •

Researchers exploring a new area;

• •

Researchers writing papers for peer review, to ensure reviewers will not reject a paper since it omits important related work;

• •

Researchers conducting a systematic literature review on a specific field to learn about and summarize the latest developments.

Literature reviews can be extremely labor intensive due to the low prevalence of relevant papers. Here in this paper, we take the example of systematic literature reviews to demonstrate this problem. For example, systematic literature reviews are the primary method for aggregating evidence in evidence-based software engineering [25]. In such reviews, researchers thoroughly analyze all the research papers they can find to synthesize answers to some specific research questions. One specific step in systematic literature review, which is called primary study selection, is to find most (if not all) of the relevant papers to the research questions. This step is identified as one of the most difficult and time consuming steps in systematic literature review [3] as it requires humans to read and classify thousands of candidate papers from a search result. Table 1 shows four systematic literature reviews where thousands of papers were reviewed before revealing just a few dozen relevant papers. Shemilt et al., [43] estimated that assessing one paper for relevancy takes at least one minute. Assuming 25 hours per week for this onerous task, the studies of Table 1 would take 16 weeks to complete (in total). Therefore, reducing the human efforts required in this primary study selection step is thus critical for enabling researchers conducting systematic literature reviews more frequently.

Many researchers have tried reducing the effort associated with literature reviews [31, 2, 23, 55, 35, 37, 7, 1, 30, 41]. Prior work from the authors [57] focused on retrieving most relevant papers with least effort and found that active learning is the best way to achieve that. In the state of the art active learning approaches [8, 54, 34], a support vector machine (SVM) is updated whenever a human reviewer decided if a paper is relevant/non-relevant to a specific research question. The active learner then reflect over the updated SVM to select which paper is most informative to read next. In this active learning manner, human review efforts are focused on the papers that either is most likely to be relevant or can improve the current classifier most. Furthermore, advantages from different active learning approaches are adopted to derive a new approach FASTREAD [57], which outperformed the previous state of the art approaches [8, 54, 34].

While a useful tool, FASTREAD does not address three important research questions (that are resolved in this paper):

RQ1: “How to start?”; i.e., how to control initial paper selection. The resulting incremental learners can vary wildly, depending on the initial selection of examples. This is important since, as shown below in Section 2.5, a poor initial selection of papers can greatly increase the number of papers that must be read. But as shown in this paper:

{bclogo}

[couleur=gray!20,arrondi=0.1,logo=\bclampe,ombre=true] How to start? The first relevant paper can be identified earlier, robustly, by applying a little domain knowledge to guide the initial sampling.

RQ2: “When to stop?”; i.e., how to know when can the review be safely stopped. While there is probably always one more relevant paper to find, it would be useful to know when most papers have been found (say, 95% of them [6]). Without the knowledge of how many relevant papers that are left to be found, researchers might either:

• •

Stop too early, thus missing many relevant papers.

• •

Stop too late, causing unnecessary further reading, even after all relevant items are find.

This paper shows that the rate at which incremental SVM tools finds papers follow a simple mathematical relationship. Hence:

{bclogo}

[couleur=gray!20,arrondi=0.1,logo=\bclampe,ombre=true] “When to Stop?” During the review process, the current achieved recall of relevant papers can be accurately estimated through a semi-supervised logistic regressor. Therefore it is possible to determine when a target recall (say, 95%), has been reached.

We show, in Section 4.4 that our stopping rule is more accurate than other state of the art stopping criteria [49].

RQ3: “How to correct?”. Human reviewers are not perfect, and sometimes they will label relevant papers as non-relevant, and vice versa. Practical tools for support literature reviews must be able to recognize and repair incorrect labeling. We show here that:

{bclogo}

[couleur=gray!20,arrondi=0.1,logo=\bclampe,ombre=true] How to correct? The human labeling errors can be efficiently identified and corrected by periodically rechecking few of the labeled papers, whose labels the active learner disagrees most on.

This paper is structured as follows. The rest of this section states our contributions; connections to prior work; and some caveats. After that, background notes are presented selecting relevant papers in Section 2, techniques and experimental results for the how to start problem in Section 3, answering to the when to stop problem in Section 4, and results for How to correct problem when human errors are taken into consideration in Section 5. Section 6 describes the overall tool FAST2 as a combination of all the techniques from Section 3 to Section 5 and discusses the threats to validity. Conclusions are provided in Section 7.

Note that all our case studies are systematic literature reviews from the field of software engineering. We use that data since that is our “home” domain and it seems more responsible to draw data from an area that we know most about. That said, there is nothing in principle stopping the reader from applying these methods to their own home domains. To that end, we offer a full reproduction package with all this study’s code and data [58].

1.1 Contributions

This paper offers:

A starting tactic for literature reviews to find the first relevant paper faster, robustly. 2. 2.

An early stopping rule which, for our test data, is far more accurate than other state of the art stopping criteria. 3. 3.

An error correcting strategy that mitigates human errors in literature reviews. 4. 4.

A software tool called FAST2 that implements the above three points, thus addressing three open issues in prior fast reading tools, i.e. how to start, when to stop, how to correct. 5. 5.

A reproduction package with all this study’s code and data [58].

1.2 Connection to Prior Work

This paper is a significant improvement over FASTREAD, the tool proposed in our prior work [57] on automatic support for literature reviews. In this prior work [57], the authors

Analyzed the shortcomings of existing systematic review tools, e.g. Abstrakr 111http://abstrackr.cebm.brown.edu [51], EPPI-Reviewer 222http://eppi.ioe.ac.uk/cms/er4/ [44], Rayaan 333http://rayyan.qcri.org/ [36]. 2. 2.

Compared different approaches, such as search-query based methods [59, 45], reference-based methods [23, 16, 55], supervised learning [7, 1], semi-supervised learning [30], unsupervised learning [31], and found that active learning is the most efficient in reducing the cost of primary study selection. 3. 3.

Constructed three software engineering systematic literature review datasets by reverse engineering from the original publications. 4. 4.

Mixed and matched the state of the art active learning approaches [54, 34, 8] to generate 32 possible combinations. 5. 5.

The resulting best method FASTREAD had top rank performance across all datasets and significantly outperformed the three existing state of the art methods.

More details on FASTREAD will be provided in Section 2.4. While FASTREAD was the state of the art approach for selecting which papers to read next, it offered no support for the RQ1, RQ2, RQ3 issues that are solved in this paper.

1.3 Caveats

One caveat for the following results is that all our test data comes from the software engineering literature. We constrained ourselves to that domain since we have personnel contacts with many researchers performing literature reviews in that domain. This constraint had many advantages, including our ability to access more test data. Note that there is nothing in principle from applying our methods to other domains.

Another important caveat is that we say that we are “reducing the effort of literature reviews”, we mean “reduce the effort of the initial skim of the papers”. Formally, in the terminology of Kitchenham’s systematic literature review (SLR) [26] framework, this “initial skim” is the primary study selection phase of a literature review. We focus on this phase of the literature review process for two reasons:

• •

Prior work by Carver and others [3, 22, 21] showed that primary study selection is one of the slowest parts of the entire literature review process, as shown in Figure 1.

• •

There are well-established tools to handle much of the rest of the SLR process. From example, Marshall et al., [32] describe tools to assist in searching for papers then extracting papers from multiple databases and assessing their “quality” (i.e. their relevance to the query). But a large missing piece in the current toolchain for SLRs are supports for selecting a small number of relevant papers from within the larger set of papers found by searching the databases.

Another important caveat is that we are careful to say that our methods can find many relevant papers, rather than all relevant papers. Our goal is to offer engineering principles for literature reviews, and engineering is the discipline that delivers acceptable products in reasonable time. Hence, like many other researchers in this area, our success criteria is “work saved oversampling at 95% recall” (WSS@95) [6]. If researchers hope to find all relevant papers, then they should (a) ignore the methods of this paper and (b) allocate a very long time to their literature reviews.

2 Background

2.1 Literature Reviews in Different Domains

Selecting which technical papers to read is a task that is relevant and useful for many domains. For example:

• •

In legal reasoning, attorneys are paid to review millions of documents trying to find evidence to some case. Tools to support this process are referred to as electronic discovery [19, 8, 9, 11].

• •

In evidence-based medicine, researchers review medical publications to gather evidence for support of a certain medical practice or phenomenon. The selection of related medical publications among thousands of candidates returned by some search string is called citation screening. Text mining methods are also applied to reduce the review effort in citation screening [34, 54, 52, 53, 49, 51, 50].

• •

In software engineering, Kitchenham et al. recommend systematic literature reviews (SLRs) to be standard procedure in research [28, 25]. In systematic literature reviews, the process of selecting relevant papers is referred to as primary study selection when SE researchers review titles, abstracts, sometimes full texts of candidate research papers to find the ones that are relevant to their research questions [26, 25].

For the most part, SLRs in SE are a mostly manual process (evidence: none of the researchers surveyed in Fig 1 used any automatic tools to reduce the search space for their reading). Manual SLRs can be too slow to complete or repeat. Hence, the rest of this paper explores methods to reduce the effort associated with literature reviews.

2.2 Automatic Tool Support

Formally, the problem explored in this paper can be expressed using the nomenclature of Table 2:

• •

Starting with $L=\emptyset$ papers;

• •

Prioritize which papers to be reviewed so as to…

• •

Maximize $|L_{R}|$ (the number of relevant papers discovered)…

• •

While minimizing $|L|$ (the number of papers reviewed).

Tools that address this problem can be divided as follows:

Search-query based that involves query expansion/rewriting based on user feedback [45]. This is the process performed by any user as they run one query and reflect on the relevancy/irrelevancy of the papers returned by (e.g.) a Google Scholar query. In this search-query based approach, users struggle to rewrite their query in order find fewer non-relevant papers. While a simple method to manually rewrite search queries, its performance in selecting relevant papers is outperformed by the abstract-based methods described below [18, 24].

Reference based methods such as “snowballing” exploit citation links between research papers. Starting with papers known to be relevant, “forward” and “backward” snowballing chases new relevant papers through papers cite the known relevant ones and references of known relevant ones, respectively [23, 16, 55]. While a straightforward method, reference-based methods suffer from two disadvantages—a) the cost for extracting references and forming a citation link graph ( full papers have to be investigated before they go into the snowballing procedure [55]); b) precision and recall vary a lot across different studies (Wohlin reported in one of his studies as precision to be 6.8% [55] while in another study as precision to be $15/64=23.4\%$ [23]).

Abstract based methods use text from a paper’s abstract to train text classification models, then apply those models to support the selection of other relevant papers. Since it is easy to implement and performs reliably well, abstract-based methods are widely used in many domains [57, 34, 54, 8].

2.3 Abstract-based Methods

In this paper, we focus on abstract-based methods since:

• •

They are easy to implement—the only extra cost is some negligible training time, compared to search query based methods which require human judgments on keywords selection and reference based methods which cost time and effort in extracting reference information.

• •

Several studies show that the performance of abstract-based methods is better than other approaches [24, 40].

Note that abstract-based methods do not restrict the reviewers from making decisions based on full-text when it is hard to tell whether one paper should be included or not based on abstract. However, only abstracts are used for training and prediction, which makes data collection much easier. Abstract-based methods utilize different types of machine learning algorithms :

Supervised learning: which trains on labeled papers (that humans have already labeled as relevant/non-relevant) before classifying the remaining unlabeled papers automatically [7, 1]. One problem with supervised learning methods is that they need a sufficiently large labeled training set to operate and it is extremely inefficient to collect such labeled data via random sampling. For this reason, supervised learners are often combined with active learning (see below) to reduce the cost of data collection. 2. 2.

Unsupervised learning: techniques like visual text mining (VTM) can be applied to facilitate the human labeling process [31, 17]. In practice the cost reductions associated with (say) visual text mining is not as significant as that of active learning methods due to not utilizing any labeling information [40, 24]. 3. 3.

Semi-supervised learning: are similar to supervised learning but utilizes unlabeled data to guide the training process. Different from active learning, it does not actively query new labels from unlabeled data. Instead, it tries to make use of the unlabeled data to improve the training result of a supervised learning [4]. Semi-supervised methods have been proved to be more effective than supervised learning methods but still suffers from the same problem (the need for relatively large labeled training sets–Liu et al., [30] reported a less than 95% recall with 30% data labeled). 4. 4.

Active learning: In this approach, human reviewers read a few papers and classify each one as relevant or non-relevant. Machine learners then use this feedback to learn their models incrementally. These models are then used to sort the stream of papers such that humans read the most informative ones first.

2.4 FASTREAD

Our prior work in this area leads to the FASTREAD abstract-based active learner [57] shown in Algorithm 1. FASTREAD employed the following active learning strategies:

• •

Input tens of thousands of papers selected by (e.g.) a query to Google Scholar using techniques such as query expansion [45].

• •

In the initial random reasoning phase (shown in line 9 of Algorithm 1),

• –

Allow humans to skim through the papers manually until they have found $|L_{R}|\geq 1$ relevant paper (along with $|L_{I}|$ non-relevant ones). Typically, dozens of papers need to be examined to find the first relevant paper.

• •

After finding the first relevant paper, transit to the reflective reasoning phase (shown in line 6 to 7 of Algorithm 1):

• –

Train an SVM model on the $L$ examples. When $|L_{R}|\geq 30$, balance the data via aggressive under-sampling; i.e., reject all the non-relevant examples except the $|L_{R}|$ non-relevant examples furthest to the decision plane and on the non-relevant side.

• –

Use this model to decide what paper humans should read next. Specifically, find the paper with the highest uncertainty (uncertainty sampling) when $|L_{R}|<30$ (shown in line 25 of Algorithm 1) and find the paper with highest probability to be relevant (certainty sampling) when $|L_{R}|\geq 30$ (shown in line 27 of Algorithm 1).

• –

Iterate the reflective reasoning phase until more than 95% of the relevant papers have been found ($|L_{R}|\geq 0.95|R|$).

FASTREAD was designed after testing and comparing 32 different active learners generated from three prior state of the art methods from the medical and legal literature—Wallace’10 [54], Miwa’14 [34], and Cormack’14 [8].

• •

When compared to manual reading methods (linear review), FASTREAD dramatically reduces the effort associated with studies like those in Table 1. With the help of FASTREAD, an order of magnitude fewer papers are required to be reviewed to find 95% of the relevant ones.

• •

When compared to current state of the art automatic methods—Wallace’10 [54], Miwa’14 [34], and Cormack’14 [8], FASTREAD reviews up to 50% fewer papers while finding the same number of relevant papers.

• •

When compared to all the 32 methods used to design FASTREAD, it performed remarkably better (see Figure 2).

2.5 Limitations with FASTREAD

While successful in some aspects, FASTREAD suffers from three significant limitations.

Firstly, FASTREAD utilizes no domain knowledge and knows nothing about its retrieval target before human oracles are provided. Hence, the order of the papers explored in the initial reasoning phase is pure guesswork. This can lead to highly undesirable results, as shown in Figure 3:

• •

In that figure, the y-axis shows what percentage of the relevant papers were found (the data here comes from the Hall dataset in Table 1).

• •

The lines on this figure show the median (green) and worst-case (red) results in 30 runs, we varied, at random, the papers used during initial reasoning.

• •

The dotted and solid lines show the performance of FASTREAD and FAST2 respectively.

Observe from the curves in Figure 3:

• •

The solid curves denote FASTREAD’s performance, and the distance between FASTREAD’s median and worst case is a factor of three. This means the random selections made during the initial phase can have runaway effects; i.e., an undesirable effect of greatly increasing the effort associated with literature reviews.

• •

On the other hand, as shown by the dashed curves of Figure 3, FAST2 has little variance and does not suffer from that runaway effects. As shown later in this paper, FAST2 achieves this via requesting minimal amounts of domain knowledge.

Secondly, using FASTREAD reviewers do not know when to stop reviewing. FASTREAD has no estimator that users can query to ask “is it worth reading any more papers?”. Without such knowledge of how many relevant papers that are left to be found, researchers might either:

• •

Stop too early, thus missing many relevant papers;

• •

Or stop too late, causing unnecessary effort even when there are no more relevant papers to find.

The dashed lines in Figure 3 show how FAST2 stops near 0.95 recall without knowing the actual number of relevant papers to be found ($|R|$). Section 4 provides details about how FAST2 achieves this.

Thirdly, FASTREAD assumes reviewers are infallible; i.e., when labeling papers as relevant or non-relevant, they never make mistakes. This seems a very optimistic assumption. In practice, human errors are inevitable so any intelligent reading assistant should have some mechanism for handling those errors. Figure 4 simulates a literature review on Hall dataset with a fallible human reviewer whose precision and recall are both 70% (i.e., when asking the reviewer to label all papers, 70% recall, and 70% precision can be achieved). This is a reasonable assumption according to Cormack and Grossman, [12]. Solid lines show the performance without any error correction while dashed lines show the improved results (with increasing true positives and decreasing false negatives and false positives) with error correction method in FAST2. Details of the error correction method in FAST2 will be provided in Section 5.

3 How to Start

The rest of this paper explores ways to resolve the limitations of FASTREAD. In this first section, we drop the assumption of no domain knowledge to answer the following research question:

RQ1: How to start? How to better utilize external domain knowledge to boost the review process and avoid the runaway effects illustrated in Figure 3?

3.1 Related Work

Applying domain knowledge to guide the initial selection of training data, instead of random sampling, can avoid efforts being wasted on non-relevant examples and prevents runaway results. However, no matter what form of domain knowledge applied, a common issue will be raised when random sampling is replaced; i.e., how to collect unbiased non-relevant examples. Without random sampling, the training data collected by domain knowledge guided review, uncertainty sampling, or certainty sampling are all biased and lead to a deteriorated performance when used for training.

Our solution to this problem is based on presumptive non-relevant examples (shortened as PRESUME in the rest of this paper) proposed in the legal literature by Cormack and Grossman, [9]. Each time before training, PRESUME samples randomly from the unlabeled examples and presumes the sampled examples to be non-relevant in training. The rationale behind this technique is that given the low prevalence of relevant examples, it is likely that most of the presumed ones are non-relevant.

With the help of PRESUME, Cormack and Grossman, [9] investigated two different ways to utilize different forms of domain knowledge for a better collection of initial training data:

• •

Auto-BM25: keyword search and rank the unlabeled examples with their corresponding BM25 scores, which is calculated based on tf-idf [39]. Then label the example with highest BM25 score as relevant and start training. Given a search query $Q=\{q_{1},q_{2},\dots,q_{n}\}$, the BM25 score is calculated as

[TABLE]

where $f(q_{j},i)$ is term $q_{j}$’s term frequency in document $i$, $l_{i}$ is the length of document $i$, and $avgdl$ is the average document length in the collection $C$. $k_{1}$ and $b$ are free variables, here, we choose $K_{1}=1.5$, $b=0.75$. $IDF(q_{j})$ is the inverse document frequency weight of term $q_{j}$ and it is computed as:

[TABLE]

where $n(q_{j})$ is the number of documents containing term $q_{j}$.

• •

Auto-Syn: a synthetic document is created, with domain knowledge from the reviewer, to act as the first relevant example and train the classifier directly.

3.2 Method

While the above methods from Cormack and Grossman, [9] were shown to be useful in their test domain, they have certain limitations. Both methods rely on domain knowledge from reviewers to generate one seed relevant example and then use it to initiate the active learning process. One problem with such techniques is that the quality of the seed example, which is decided by the reviewers’ expertise, can affect the performance of the active learner. More efforts are required to find new relevant examples if the seed example is not representative enough.

To address this concern, in our work we modified Auto-BM25 and created a new method called Rank-BM25:

• •

Just as with Auto-BM25, Rank-BM25 starts with querying a set of keywords and rank the unlabeled examples based on their BM25 scores.

• •

Rank-BM25 then asks a reviewer to review $N=10$ examples with descending order of their BM25 scores.

• •

If $|L_{R}|\geq 1$ relevant examples found, start active learning; otherwise Rank-BM25 tries a different keyword set.

3.3 Experiments

In this subsection, we design experiments to answer RQ1 by comparing different ways of utilizing external domain knowledge using the methods described above.

3.3.1 Datasets

Previously, we have created three datasets by reverse-engineering existing SLRs [57]. The three datasets are named after the authors of their original publication source—Wahono dataset from Wahono, [47], Hall dataset from Hall et al., [20], and Radjenović dataset from Radjenović et al., [38]. Apart from the three created datasets, one dataset (Kitchenham) is provided directly by the author of Kitchenham and Brereton, [27]. Statistics of the four datasets are shown in Table 1. In this paper, we use the same four datasets to evaluate different techniques. All the above datasets are available online [58].

It is appropriate to ask why we used these four datasets and not four others? The answer is that, in the domain of automated support for reading the SE literature, there is very little data available for experimentation. We had to build three of the data sets from the last paragraph—a process that took weeks of work. Only after extensively advertising our technique did we receive offers of other data from other researchers (and that lead to the four datasets contributed by Kitchenham).

3.3.2 Performance Metrics

In order to compare with FASTREAD, the same performance metrics are applied here:

• •

X95 = $\min\{|L|\mid|L_{R}|\geq 0.95|R|\}$.

• •

WSS@95 = $0.95-\text{X95}/|P|$.

X95 stands for the number of papers need to be reviewed in order to get $0.95$ recall, while WSS@95 represents the work saved, comparing to random sampling, when reaching $0.95$ recall. The reason behind $0.95$ recall is that a) $1.00$ recall can never be guaranteed by any text mining method unless all the candidate papers are reviewed; b) $0.95$ recall is usually considered acceptable in evidence-based medicine [6, 7, 35] despite the fact that there might still be relevant papers missing [43].

As recommended by Mittas & Angelis in their 2013 IEEE TSE paper [33], Scott-Knott analysis was applied to cluster and rank the performance (X95) of each treatment. It clusters treatments with little difference in performance together and ranks each cluster with the median performances [42]. As suggested in the authors’ prior work [57], nonparametric hypothesis tests are applied to handle the non-normal distribution. Specifically, Scott-Knott decided two methods are not of little difference if both bootstrapping [14], and an effect size test [5] agreed that the difference is statistically significant ($99\%$ confidence) and not a negligible effect (Cliff’s Delta $\geq$ 0.147).

3.3.3 Treatments

As for RQ1: “How to start?” four different treatments are tested, including the random sampling tactic from FASTREAD [57] as a baseline, Auto-BM25 and Auto-Syn from Cormack and Grossman, [9], and Rank-BM25 created in this paper. To compare performance and variance of different treatments, each one is tested on all four datasets with 30 random seeds. For each treatment, the corresponding domain knowledge imported is as following:

• •

Rank-BM25: “Topic” column in Table 1 as the search query $Q$.

• •

Auto-BM25: “Topic” column in Table 1 as the search query $Q$.

• •

Auto-Syn: “Original Title” column in Table 1 as the synthetic document.

• •

FASTREAD: random sampling until the first relevant paper is found, same as FASTREAD.

Note that for all treatments, PRESUME is required to keep the training data unbiased.

3.4 Results

Table 3 shows the results of different starting tactics on four SE SLR datasets. Medians and IQRs (75-25th percentile, smaller IQR means less variances) are shown in this table. On each dataset, different treatments are compared by their X95 and WSS@95 scores (X95 smaller the better, WSS@95 larger the better). Scott-Knott analysis is applied to rank different treatments in “Rank” column. Treatments ranked the same by Scott-Knott analysis are considered to have similar performance while treatments ranked differently are significantly different in performance. With the results in Table 3, we reach the following conclusions.

Rank-BM25 is recommended as the overall best tactic. It is the only tactic which consistently performs better than the baseline tactic FASTREAD (top rank in 3 out of 4 datasets, second rank in the other dataset). While this tactic requires two or three keywords to initiate, they can be easily found in the original search string.

Auto-BM25 is considered as a depreciated version of Rank-BM25 since it never ranks better than Rank-BM25 (worse in three and similarly in one dataset).

Auto-Syn is not recommended. It requires human experts to synthesize example relevant text, which is a difficult task. The quality of the synthesized example may affect the performance a lot, thus leads to unstable results (it performs the best in two datasets but also performs worse, though not significantly, than the baseline in one dataset).

FASTREAD is not recommended since domain knowledge as trivial as two or three keywords can save 10-20% more review efforts and reduce the IQRs greatly (e.g., IQRs of Rank-BM25 are $30-50\%$ of those of Random). This indicates better robustness of the active learner and runaway results (where some readings take far longer than others due to random selection of initial papers) are far less likely when applying domain knowledge to start the review.

Hence we say:

{bclogo}

[couleur=gray!20,arrondi=0.1,logo=\bclampe,ombre=true] Answer to RQ1 “How to Start?” Overall, we suggest Rank-BM25 as the most effective starting tactic for initiating active learning. By applying such tactic, the review effort can be reduced by 10-20% by FASTREAD while preventing the runaway effects.

The updated algorithm with Rank-BM25 is shown in Algorithm 2. BM25 score is calculated as (1). Rows colored in yellow are the differences from Algorithm 1. Functions already described in Algorithm 1 are omitted.

4 When to Stop

This section drops the assumption of reviewer knows when to stop reviewing on top of Algorithm 2 and answers the following research question:

RQ2: When to stop? Without knowing the actual number of relevant papers in the candidate set, how to decide when to stop reviewing so that 1) most relevant papers have been retrieved, 2) not too much review effort is wasted.

4.1 Related Work

When to stop is a critical problem while applying active learning on literature reviews. If the reading stops too early, it may end up with missing too many relevant papers. On the other hand, if the reading stops too late, review effort might be wasted since no new relevant papers can be found.

Before this study, other work [54, 34] usually focus on generating the best reading curve and do not discuss the stopping rule. That said, our reading of the literature is that there exists three candidate state of the art stopping rules for selecting relevant papers in software engineering, electronic discovery, and evidence-based medicine respectively:

• •

Ros’17: stop review after 50 non-relevant papers are found in succession [41].

• •

Cormack’16: the knee method [10]. This method detects the inflection point $i$ of current recall curve, and compare the slopes before and after $i$. If $slope_{<i}/slope_{>i}$ is greater than a specific threshold $\rho$, the review should be stopped. For details about this knee method, please refer to Cormack and Grossman, 2016a [10].

• •

Wallace’13: Apply an estimator to estimate the number of relevant papers $|R|$ and let the users decide when to stop by showing them how close they are to the estimated number [49].

We note that the Ros’17 and Cormack’16 methods are less flexible than the Wallace’13 method since they do not allow the user to choose what recall they want. Also note that another recently proposed stopping strategy by Di Nunzio, [13] requires the review to follow the order of BM25, therefore is not compared in this papers.

4.2 Method

According to Cohen, [6], we want the reading to stop at 95% recall. To achieve this, we must first know the total number of relevant papers $|R|$. Since this is unknowable in practice, the “when to stop” problem can be solved by building a good class probability estimator which can tell the reviewer a) what is the probability of being relevant for each unlabeled paper; b) how many relevant papers have not been found yet.

The challenges for building such a class probability estimator include the class imbalance of training data [48] and the bias incurred by sampling with active learning. Specifically, given that active learning provides a better chance to retrieve relevant papers in an early stage, the prevalence of relevant papers in the reviewed set $P_{L}=|L_{R}|/|L|$ is much larger than that in the candidate set $P=|R|/|E|$ [49].

Wallace’13 solved the above challenges by sampling $\propto$ probabilities—sample unlabeled examples with a probability of active learner’s prediction. However, sampling $\propto$ probabilities is less efficient than the query strategy of FASTREAD (details will be presented in Section 4.4). Therefore we design a new estimator SEMI which utilizes the same query strategy of FASTREAD and fits a semi-supervised logistic regressor utilizing not only the labeled data but also the unlabeled data.

SEMI utilizes a recursive TemporaryLabel technique. Each time the SVM model is retrained, SEMI assigns temporary labels to unlabeled data points and builds a logistic regression model on the temporary labeled data. It then uses the obtained regression model to predict on the unlabeled data and updates the temporary labels. This process is iterated until convergence. Algorithm 3 shows how SEMI works in detail where the LogisticRegression function is implemented with scikit-learn, and its regularization strength $C$ is calculated as $C=|R_{E}|/(|L|-|L_{R}|)$ to mitigate the imbalance in SVM classifier.

4.3 Experiments

In this subsection, we design experiments to answer RQ2 by 1) comparing the estimator of SEMI with Wallace’13, 2) comparing the performance of applying SEMI as stopping rule with Ros’17 and Cormack’16 stopping rules.

Firstly, SEMI was compared with the Wallace’13 estimators for the task of estimating the prevalence of relevant examples ($|R|/|E|$). Results from uniform random sampling 444Uniform random sampling samples from unlabeled data randomly and queries the labels, it then estimates $|R|$ as $|R_{E}|=|E|\times|L_{R}|/|L|$. It was observed to estimate more accurately than the Wallace’13 estimator according to Wallace et al., 2013a [49]. is also provided as a baseline.

We then used the SEMI estimator as an early stopping rule for literature reviews. Reviews were simulated on our four full datasets and stopped when $|L_{R}|\geq T_{rec}|R_{E}|$. $T_{rec}$ represents the desired final recall of the review and $T_{rec}=0.90$ for SEMI (90), $T_{rec}=0.95$ for SEMI (95). Note that $R_{E}$ was calculated each time the learner was retrained. Performances of this new stopping rule were analyzed by comparing with :

• •

Ros’17: i.e., stopping when 50 non-relevant papers found in succession;

• •

Cormack’16: The knee method with $\rho=6$.

Note that besides $\rho=6$, Cormack and Grossman, 2016a [10] have suggested an alternative $\rho=156-min(|L_{R}|,150)$, especially for the cases of a relatively small dataset. However when we tested it, Cormack’16 stopping rule barely worked with $\rho=156-min(|L_{R}|,150)$ and almost all candidate papers were reviewed. Therefore we choose $\rho=6$ for the Cormack’16 method.

4.4 Results

Before applying SEMI as a stopping rule, the accuracy of its estimation is first tested against that of the prior state of the art estimator—the Wallace’13 estimator. Figure 5 shows the performances of different estimators. The first column in Figure 5 demonstrates the effectiveness of different query strategies. Suggested by these results, SEMI (in blue) outperforms Wallace’13 (in green) and uniform random sampling (in red) since it achieves same recall with much fewer papers reviewed.

The second column of Figure 5 demonstrates the effectiveness of different estimators. The SEMI estimator (in blue) is not only more accurate than the Wallace’13 estimator (in green) but also the estimation from uniform random sampling (in red). SEMI is a better estimator than the state of the art Wallace’13 estimator since both its query strategy and its estimation are better. Moreover, in all our case studies, after reviewing 150 documents, the SEMI estimator becomes remarkably accurate. This means that, very early in a literature review, researchers can adjust their review plans according to the number of remaining papers.

Then we apply SEMI as the stopping rule and compare it against other state of the art stopping rules. Based on the results shown in Table 4, we observe that Ros’17 usually stops too early and results in a low recall; Cormack’16 results in a large range of recalls which the user has no control of. On the other hand, we suggest SEMI as the best stopping rule since (1) it has the advantage of letting user to choose the target recall ($T_{rec}$) and (2) based on the results, it stops very close to the desired recall stably (with lower IQRs).

To summarize this section:

{bclogo}

[couleur=gray!20,arrondi=0.1,logo=\bclampe,ombre=true] Answer to RQ2 “When to stop?” With our proposed estimator SEMI, we suggest the review to stop when $|L_{R}|\geq T_{rec}|R_{E}|$. We have shown in our results that the proposed stopping rule can stop the review close to the target recall.

The updated algorithm with SEMI as stopping rule is shown in Algorithm 3. Rows colored in yellow are the differences from Algorithm 2. Functions already described in Algorithm 2 and 3 are omitted.

5 How to correct

This section drops the assumption of Reviewer never make mistakes on top of Algorithm 3 and explores this question:

RQ3: How to correct? Human makes mistakes, and those mistakes can be amplified by misleading the machine learning model to train on the wrongly labeled papers. How to correct human error in time and accurately is a huge challenge.

5.1 Related Work

The observation of human errors in literature review is prevalent, e.g., Wohlin et al., [56] documented that reviewers will not find same papers even with thorough reviews, and Cormack and Grossman, [12] concluded that a human reviewer could achieve on the order of 70% recall and 70% precision. The human error becomes an even more severe problem when active learning is applied to assist the review since the active learning model would be misled by the mislabeled training examples [46]. Therefore correcting these human errors becomes a crucial task for this work.

Via our literature review, we found two state of the art error correction methods that can be applied to solve this problem:

• •

Kuhrmann’17: One simple way to correct human errors is by majority vote. As Kuhrmann et al., [29] described in 2017, simplest form of majority vote requires every paper to be reviewed by two different reviewers, and when the two reviewers do not label it consistently, a third reviewer is asked to make the final decision. The advantage of this method is that the human errors are corrected immediately and will not mislead the active learning model. However, this method requires at least three different reviewers and will at least double the review effort.

• •

Cormack’17: This error correction method [12] is built upon the Cormack’16 knee stopping rule [10]. After the review stops, papers labeled as “relevant” but reviewed after the inflection point ($x>i$) and papers labeled as “non-relevant” but reviewed before the inflection point ($x<i$) are sent to reviewers for a recheck. Comparing to Kuhrmann’17, this method requires much less extra review effort on rechecking, but can only correct human errors after the review, which leads to bias in the active learning model.

5.2 Method

Taking into consideration the advantages and disadvantages of the existing error correction methods, we utilize the class probability estimation from SEMI to build a new error correction method called Disagree which now and then (whenever 50 new papers are labeled) rechecks some of the labeled papers. To improve the efficiency of the error correction, only those papers with labels that the current active learner disagrees most on are rechecked. Since human error rate on mislabeling relevant as non-relevant is much higher than vice versa (thus creating say 70% precision and 70% recall on highly imbalance dataset), rechecking effort is focused more on papers labeled as non-relevant.

In this way, Disagree can 1) correct human errors in time; and 2) avoid wasting too much effort on rechecking correctly labeled papers. Algorithm 4 shows how Disagree works in detail where the IncludeError function simulates a human reviewer with a precision of $Prec$ and recall of $Rec$.

5.3 Experiments

In this subsection, we design experiments to answer RQ3 by comparing Disagree with:

• •

None: i.e. no error correction (so just Algorithm 3 with human errors);

• •

Kuhrmann’17: majority vote with three reviewers of same error rate; and

• •

Cormack’17: knee method with $\rho=6$.

Our simulations are conducted on the four SE literature review datasets of Table 1 with increasing human error rate injected (from 100% recall and 100% precision to 70% recall and 70% precision).

Because of the introduction of human errors, $L_{R}\neq L\cap R$ anymore. We use $tp=|R\cap L_{R}|$ to represent the true positive, and $ct$ to represent the actual review effort ($ct\leftarrow ct+1$ each time a new/labeled paper is reviewed/rechecked). Performance of each method is assessed by the following metrics:

• •

Recall: $tp/|R|$ measures the percentage of relevant papers being included in the final inclusion list. The Higher, the better.

• •

Precision: $tp/|L_{R}|$ measures that in the set of papers labeled as relevant by human reviewers, how many of them are relevant. The higher, the better.

• •

Effort: $ct$ measures the actual number of papers being reviewed by human reviewers (including rechecking). The lower, the better.

5.4 Results

Table 5 shows the median results for 30 repeated simulations. While we collected results for human reviewer recall, precision $\in\{100\%,90\%,80\%,70\%\}$, the trend across all the results was apparent. Hence, for the sake of brevity, we just discuss the 100% and 70% results. Full results are available at https://tiny.cc/error_fast2. Note that:

• •

For precision and recall, better tactics have less deterioration as errors increase; i.e. higher b/a and d/c values are better.

• •

For effort (which measures the number of papers reviewed/rechecked), better tactics demand fewer papers are read; i.e. lower f/e values are better.

• •

The cells in gray show median changes over the four data sets for all the error correction tactics.

Of the results in Table 5, the None tactic (which does not feature any error correction) is the worst. While it paints good precision, as errors are added to the labeling process, recall decreases to 69% and 53% more efforts need to be spent on reviewing.

Kuhrmann’17 is the go-to treatment if 100% precision is required. The downside is that it usually requires most effort to achieve the 100% precision and its recall is second worst among all four treatments. In general, we do not suggest Kuhrmann’17 since recall is most important.

Cormack’17 is the least stable treatment among the four. On Wahono, Hall, Radjenović datasets, it uses the least effort but achieves very poor recall; while on Kitchenham dataset, it achieves highest recall but also costs most effort.

Turning now to the Disagree results, we see that this tactic maintains high recall (92% of their original values in average) with acceptable precision (91% of their original values in average) at the cost of modest increases in effort (115% of their original values in average). Comparing to the optimistic case, Disagree retains high recall when human error increases with the cost of around 100% extra effort.

{bclogo}

[couleur=gray!20,arrondi=0.1,logo=\bclampe,ombre=true] Answer to RQ3 “How to Correct?” If reviewers frequently pause and recheck old conclusions whose labels contradict the current conclusion of the learner, the performance deterioration associated with human error can be alleviated without much overhead rechecking effort.

The updated algorithm with Disagree to correct human misclassifications is shown in Algorithm 4. Rows colored in yellow are the differences from Algorithm 3. Functions already described in Algorithm 3 are omitted.

6 Discussion

6.1 FAST2

Combining the results above from Section 3 to Section 5, we propose our new reading tool FAST2 which is built on top of FASTREAD [57] and its framework is shown in Figure 6 with the following features:

• •

Initial Sampling with BM25 ranking keyword search.

• •

Recall Estimation by SEMI estimator.

• •

Error Prediction by Disagree.

When applied to other domains, same framework should be followed while detailed techniques can be changed or adapted. The new FAST2 tool is available on Zenodo [58] and is continuously updated on Github 555https://github.com/fastread/src.

6.2 Threats to Validity

There are several validity threats to the design of this study [15]. Any conclusions made from this work must be considered with the following issues in mind:

Conclusion validity focuses on the significance of the treatment. It applies to our analysis of which treatment works best for “how to start”. Although the suggested treatment (Rank-BM25, SEMI, Disagree) performs consistently better than FASTREAD, it is not always the best treatment on every dataset.

**Internal validity **focuses on how sure we can be that the treatment caused the outcome. This applies to our analysis of which treatment works best for “how to start” since the domain knowledge applied is decided by the authors and different domain knowledge might lead to different outcomes.

**Construct validity **focuses on the relation between the theory behind the experiment and the observation. In this work, we evaluate different treatments with a target of reaching 0.95 recall. Although it is right now the widely accepted target for active learning based literature reviews [6, 35], either increasing or decreasing the required final recall may result in a different ranking of treatments.

External validity concerns how well the conclusion can be applied outside. All the conclusions in this study are drawn from the experiments running on four software engineering systematic literature review datasets generated from Hall et al., [20], Wahono, [47], Radjenović et al., [38], Kitchenham and Brereton, [27]. Therefore, such conclusions may not be applicable to datasets of different scenarios, e.g., citation screening from evidence-based medicine or TAR from e-discovery. Such bias threatens any classification experiment. The best any researcher can do is to document that bias then makes available to the general research community all the materials used in a study (with the hope that other researchers will explore similar work on different datasets). To this end, we have published all our code and data on Zenodo [58]. Also, there are parameters decided as engineering judgment which may not be the best choice. Tuning/tweaking such parameters might lead to different conclusions. Finally, this concern also applies to the results of “when to stop” and “How to correct” since the task of selecting relevant papers is multi-objective, and we choose SEMI and Disagree as for the best methods in favor of their trade-off between recall and review effort. In other circumstances, a user may prefer other methods in favor of their own goals.

7 Conclusions

Unless other people can find our research, we will risk our work being unacknowledged, uncited, and forgotten. One significant barrier to finding related research is the effort of selecting papers to read. A systematically scheduled review usually requires researchers to review thousands of papers to find the dozens of papers relevant to their research and this usually costs weeks to months of work. Previously, we have built a state of the art reading support tools to reduce that effort by applying active learning methods [57].

Although our reading tool FASTREAD achieved a good reduction on review efforts, it was not ready for practical use because of its three unrealistic assumptions: 1) no external domain knowledge, 2) reviewer knows when to stop reviewing, and 3) reviewer never make mistakes. Accordingly, in this paper, we extend our previous work by dropping and addressing the above three assumptions with different techniques either found from existing works or created by ourselves.

Our results suggested that 1) with a little domain knowledge (two or three keywords or data from previous reviews), we can further reduce the review effort by 10% to 20% with much better robustness. 2) By training a logistic regression model with semi-supervised learning, we can estimate the number of relevant papers accurately (better than the state of the art estimator Wallace’13 [49]) in an early stage, thus providing a reliable stopping rule (which is better than state of the art stopping rules in literature). 3) By asking the reviewer to recheck those papers whose labels the active learner disagrees most on, human errors can be successfully corrected without much extra review cost (proved to be a better error correction method than the state of the art ones in literature).

Based on these results, we build our new tool FAST2 as represented in Algorithm 4. It uses keywords search and BM25 ranking for finding the first relevant example, trains the SEMI estimator, rechecks labeled papers now and then, and stops review when $|L_{R}|\geq T_{rec}|R_{E}|$.

Considering the problems raised in Section 6.2, our future works will focus on the following aspects:

• •

Our conclusions are drawn from only four SE literature review datasets, which may incur sampling bias. In future work, we will validate the results on more datasets including those from medical and legal domains.

• •

Currently the target for active learning based review is to achieve 95% recall. It is an open and interesting problem whether there exists an efficient way to retrieve the rest 5% relevant papers.

• •

The magic parameters are selected based on expert suggestions. Tuning is challenging for this active learning schema since the available labeled data are limited and can only be obtained at cost. How to design a feasible tuning schema with the limited resource and how to avoid overfitting can be difficult problems but solving such problems provides great value to the domain.

Acknowledgement

The authors thank Barbara Kitchenham for her attention to this work and for sharing with us the “Kitchenham” dataset used in these experiments.