How to "DODGE" Complex Software Analytics?

TL;DR

This paper introduces DODGE, a hyperparameter optimization tool for software analytics that significantly speeds up the tuning process by avoiding redundant configurations, leading to more accurate predictive models.

Contribution

DODGE is a novel hyperparameter tuning method that efficiently skips redundant configurations, improving speed and accuracy over existing approaches.

Findings

DODGE runs orders of magnitude faster than traditional methods.

DODGE produces more accurate predictive models.

Ignoring redundant tunings enhances hyperparameter optimization efficiency.

Abstract

Machine learning techniques applied to software engineering tasks can be improved by hyperparameter optimization, i.e., automatic tools that find good settings for a learner's control parameters. We show that such hyperparameter optimization can be unnecessarily slow, particularly when the optimizers waste time exploring "redundant tunings"', i.e., pairs of tunings which lead to indistinguishable results. By ignoring redundant tunings, DODGE, a tuning tool, runs orders of magnitude faster, while also generating learners with more accurate predictions than seen in prior state-of-the-art approaches.

Tables

Table 1. TABLE I: Hyperparameter tuning options explored in this paper. Options in learners from recent SE papers on hyperparameter optimization  [ 24 , 22 , 3 , 2 ] then consulting the documentation of a widely-used data mining library (Scikit-learn  [ 49 ] ). Randint, randuniform and randchoice are all random functions to choose either integer, float, or a choice among the parameter ranges.

DATA PRE-PROCESSING
Software defect prediction: • StandardScaler • MinMaxScaler • MaxAbsScaler • RobustScaler(quantile_range=(a, b)) – a,b = randint(0,50), randint(51,100) • KernelCenterer • QuantileTransformer(n_quantiles=a, output_distribution=c, subsample=b) – a, b = randint(100, 1000), randint(1000, 1e5) – c = randchoice([‘normal’,‘uniform’]) • Normalizer(norm=a) – a = randchoice([‘l1’, ‘l2’,‘max’]) • Binarizer(threshold=a) – a = randuniform(0,100) • SMOTE(a=n_neighbors, b=n_synthetics, c=Minkowski_exponent) – a,b = randint(1,20),randchoice(50,100,200,400) – c = randuniform(0.1,5)	Text mining: • CountVectorizer(max_df=a, min_df=b) – a, b = randint(100, 1000), randint(1, 10) • TfidfVectorizer(max_df=a, min_df=b, norm=c) – a, b,c = randint(100, 1000), randint(1, 10), randchoice([‘l1’, ‘l2’, None]) • HashingVectorizer(n_features=a, norm=b) – a = randchoice([1000, 2000, 4000, 6000, 8000, 10000]) – b = randchoice([‘l1’, ‘l2’, None]) • LatentDirichletAllocation(n_components=a, doc_topic_prior=b, topic_word_prior=c, learning_decay=d, learning_offset=e,batch_size=f) – a, b, c = randint(10, 50), randuniform(0, 1), randuniform(0, 1) – d, e = randuniform(0.51, 1.0), randuniform(1, 50), – f = randchoice([150,180,210,250,300])
LEARNERS
Software defect prediction and text mining: • DecisionTreeClassifier(criterion=b, splitter=c, min_samples_split=a) – a, b, c= randuniform(0.0,1.0), randchoice([‘gini’,‘entropy’]), randchoice([‘best’,‘random’]) • RandomForestClassifier(n_estimators=a,criterion=b, min_samples_split=c) – a,b,c = randint(50, 150), randchoice([’gini’, ’entropy’]), randuniform(0.0, 1.0) • LogisticRegression(penalty=a, tol=b, C=float(c)) – a,b,c=randchoice([‘l1’,‘l2’]), randuniform(0.0,0.1), randint(1,500) • MultinomialNB(alpha=a) – a= randuniform(0.0,0.1) • KNeighborsClassifier(n_neighbors=a, weights=b, p=d, metric=c) – a, b,c = randint(2, 25), randchoice([‘uniform’, ‘distance’]), randchoice([‘minkowski’,‘chebyshev’]) – if c==’minkowski’: d= randint(1,15) else: d=2

Table 2. TABLE II: Dataset statistics. Data comes from the SEACRAFT repository: http://tiny.cc/seacraft

Dataset	No. of Documents	No. of Unique Words	Severe %
PitsA	965	155,165	39
PitsB	1650	104,052	40
PitsC	323	23,799	56
PitsD	182	15,517	92
PitsE	825	93,750	63
PitsF	744	28,620	64

Table 3. TABLE III: Top 10 topics found by LDA for PitsA dataset fromTable  II . Within each topic, the weight of words decreases exponentially left to right across the order shown here. The words here are truncated (e.g., “software” becomes “softwar”) due to stemming.

Topics=	Top words in topic
01=	command engcntrl section spacecraft unit icd tabl point referenc indic
02=	softwar command test flight srobc srup memori script telemetri link
03=	file variabl line defin messag code macro initi use redund
04=	file includ section obc issu fsw code number matrix src
05=	mode safe control state error power attitud obc reset boot
06=	function eeprom send non uplink srup control load chang support
07=	valu function cmd return list ptr curr tss line code
08=	tabl command valu data tlm load rang line count type
09=	flight sequenc link capabl spacecraft softwar provid time srvml trace
10=	line messag locat column access symbol file referenc code bld

Table 4. TABLE IV: Document Topic distribution found by LDA for PitsA dataset

Issue	10 Topics										Severe?
01	.60	.10	.00	.15	.00	.05	.03	.04	.03	.00	y
02	.10	.03	.02	.00	.03	.02	.15	.65	.00	.00	n
03	.00	.20	.05	.05	.00	.60	.02	.03	.03	.02	n
04	.03	.01	.01	.10	.15	.00	.70	.00	.00	.00	y
etc

Table 5. TABLE V: OO code metrics used for the defect prediction studies of this article. For definitions on code metrics, please refer table 1 of [ 3 ] . Last line, shown in gray , denotes the dependent variable.

amc	average method complexity
avg  cc	average McCabe
ca	afferent couplings
cam	cohesion among classes
cbm	coupling between methods
cbo	coupling between objects
ce	efferent couplings
dam	data access
dit	depth of inheritance tree
ic	inheritance coupling
lcom (lcom3)	2 measures of lack of cohesion in methods
loc	lines of code
max  cc	maximum McCabe
mfa	functional abstraction
moa	aggregation
noc	number of children
npm	number of public methods
rfc	response for a class
wmc	weighted methods per class
defects	Boolean: where defects found in bug-tracking

Table 6. TABLE VI: Statistics of the studied data sets. For the training data, the reported statistics come from the combination of all the versions used in training. In this table, the defective ratio represents the combination of total defective ratio after combining all the software versions used for training (hence, we only report one ratio.

	Training Data		Testing Data
Project	Versions	% of Defects	Versions	% of Defects
Poi	1.5, 2.0, 2.5	426/936 = 46%	3.0	281/442 = 64%
Lucene	2.0, 2.2	235/442 = 53%	2.4	203/340 = 60%
Camel	1.0, 1.2, 1.4	374/1819 = 21%	1.6	188/965 = 19%
Log4j	1.0, 1.1	71/244 = 29%	1.2	189/205 = 92%
Xerces	1.2, 1.3	140/893 = 16%	1.4	437/588 = 74%
Velocity	1.4, 1.5	289/410 = 70%	1.6	78/229 = 34%
Xalan	2.4, 2.5, 2.6	908/2411 = 38%	2.7	898/909 = 99%
Ivy	1.1, 1.4	79/352 = 22%	2.0	40/352 = 11%
Synapse	1.0, 1.1	76/379 = 20%	1.2	86/256 = 34%
Jedit	3.2,4.0, 4.1,4.2	292/1257 = 23%	4.3	11/492 = 2%

Equations

$$Recall$$

$$FPR$$

$$d2h$$

$$P_{opt}(m)=1-\frac{S(optimal)-S(m)}{S(optimal)-S(worst)}$$

$$d=smalleffect=0.2\ast \sqrt{\frac{{\sum }_i^x(x_i-(\sum x_i/n){)}^2}{n-1}}$$

Full text

1 Introduction

Fisher et al. [20] defines software analytics as a workflow that distills large quantities of low-value data into smaller sets of higher value data. Such analytics aims at generating insights and building predictors for software systems.

Due to the complexities and computational cost of SE analytics, Fisher et al warn that “the luxuries of interactivity, direct manipulation, and fast system response are gone”. In fact, they characterize modern cloud-based analytics as a throwback to the 1960s- batch processing mainframes where jobs are submitted and then analysts wait long for results with “little insight into what’s really going on behind the scenes, how long it will take, or how much it’s going to cost”. Fisher et al. document issues seen by industrial data scientists, one who says “Fast iteration is key, but incompatible with jobs … in the cloud. It’s frustrating to wait for hours, only to realize you need a slight tweak…”

One impediment to fast iterations are hyperparameter optimizers that automatically tune control options for data mining. Off-the-shelf learners come with defaults for control parameters, which may be sub-optimal. For example, in the distance function $d(x,y,p)=\left(\sum_{i}(x_{i}-y_{i})^{p}\right)^{1/p}$, a standard default is $p=2$. Yet Agrawal et al. [3] found that $p>2$ worked much better for their processing.

Hyperparameter optimizers automatically find better control parameters by experimenting with adjustments to the control parameters of a learner [10] [21]. When done using 21st century optimizers (e.g., NSGA-2 [17], IBEA [70], MOEA/D [68], FLASH [43]), it is now possible to optimize for multiple goals (even when they are competing). Table I lists some tuning options for data pre-processing and machine learning for two well-studied SE tasks:

• •

Software defect prediction (classifying modules into “buggy” or otherwise [3, 15, 22, 57, 56, 36, 24]);

• •

Software bug report text mining (to find severity [3, 44]).

Table I is a partial list of some of the tunings that might be explored. Even this incomplete sample includes billions of configuration options. With enough CPU, automatic hyperparameter optimizers can prune those options to find tunings that improve the performance of software quality predictors [3, 22, 31, 54, 56, 57, 69, 58, 44]. For example, Tantithamthavorn et al. [57, 56] showed that tuning can convert bad learners into very good ones.

The problem with hyperparameter optimization is finding enough CPU. The cost of running a data miner through all those options is very high, requiring days to weeks to decades of CPU [59, 56, 57, 62, 58, 64]. For many years, we have addressed these long CPU times via cloud-based CPU farms. Fisher et al. [20] warn that cloud computation is a heavily monetized environment that charges for all their services (storage, uploads, downloads, and CPU time). While each small part of that service is cheap, the total annual cost to an organization can be exorbitant.

Recently it was discovered how to (a) save most of that CPU cost while at the same time

(b) find better tunings. As discussed later, a method called “FFtrees” [51] (which just selects a best model within a small forest of shallow decision trees) generates much better predictions than supposed state-of-the-art results obtained after CPU intensive tuning [15]. This is strange since standard tuning tries thousands of options, but FFtrees tries just a dozen.

To explain these FFtree results [15], we observe that (a) a learner assessed by $p$ performance scores has a $p$ dimensional output space; and (b) there is some variation $\mathcal{E}$ where a learner’s performance appears in that space. As shown in Figure 1, if $\mathcal{E}=0.2$ then the 2 performance scores ($p=2$) output space divides into $1/\mathcal{E}^{p}=1/{0.2}^{2}=25$ cells. That is, if we explored more than 25 tunings, certain pairs of tunings would be redundant (i.e., would have very similar outcomes).

It turns out there are better ways to avoid redundant tunings than FFtrees. Our method DODGE($\mathcal{E}$) learns to ignore redundant tunings (parameter settings including which classifier and preprocessor to use) those that fall within $\mathcal{E}$ of other results. When tested on defect prediction and text mining, DODGE($\mathcal{E}$) terminated after fewer evaluations than standard optimizers. Also, it produced better performance scores than state-of-the art research articles (for the two well-studied SE tasks listed before [22, 3, 15, 2, 48, 24]). We conjecture that other methods perform relatively worse since they do not appreciate just how small the output space is. Hence, those other methods waste CPU as they struggle to cover billions of redundant tuning options like Table I (most of which yield indistinguishable results).

This article introduces and evaluates DODGE($\mathcal{E}$). §3 describes how FFtrees lead to the design of DODGE($\mathcal{E}$) (in §4). §5 then answers the following research questions.

RQ1: Is DODGE($\mathcal{E}$) too complicated? How to find appropriate value of $\mathcal{E}$? We can not recommend a method if it is too complex to use. Fortunately, we show that it is easy to find DODGE($\mathcal{E}$)’s parameters since its success is not altered by large changes to $\mathcal{E}$.

RQ2: How does DODGE($\mathcal{E}$) compare to recent prominent defect prediction and hyperparameter optimization results? When compared to recent tuning papers at IST’16, ICSE’18 and FSE’18 results [22, 24, 15], DODGE($\mathcal{E}$) explored a much larger parameter search space and exhibited much faster termination. Also, in terms of goal performance:

• •

DODGE($\mathcal{E}$) out-performed an ICSE’15 article exploring different learners for defect prediction [24] by around 50% and 40% for d2h and Popt(20) respectively111 d2h scores highest for models with high recalls and low false alarms while Popt(20) scores highest when many defects are localized to a small part of the code. For full details on these measures, and why we use them, see §2.3..

• •

DODGE($\mathcal{E}$) also did better than the IST’16 journal article that demonstrated the value of tuning for learners [22] by about 30% and 10% (for d2h and Popt(20)).

• •

This approach also does better than the ICSE’18 article that advocated to tune data pre-preprocessors [3] by about 10% and 5% on an average (for d2h and Popt(20)).

• •

Further, DODGE($\mathcal{E}$) also does better by 10% and 5% (for d2h and Popt(20)) than the FSE’18 article mentioned earlier that reported FFtrees [15].

RQ3: Is DODGE($\mathcal{E}$) only useful for defect prediction? In order to stress test our methods, we must apply DODGE($\mathcal{E}$) to some harder task than defect prediction. Software bug report text mining is a harder task than defect prediction since the latter only process a few dozen attributes while former task have tens of thousands of attributes. For text mining, we show DODGE($\mathcal{E}$) performs better than the IST’18 journal article that showed the value of tuning for SE text mining applications [2] by about 20% on an average for d2h. Also DODGE($\mathcal{E}$) performs better than the ICSE’13 article that applied genetic algorithms to learn the settings for a text miner [48] by about 20% on an average for d2h. As with the defect prediction studies, for both these IST’18 and ICSE’13 papers, DODGE($\mathcal{E}$) explored a much larger parameter search space and exhibited much faster termination.

From our findings, we could recommend FFtrees if the goal is only to produce succinct, approximate summaries of the factors that matter in the data. Also, as shown in Figure 2, FFtrees are slightly faster than DODGE($\mathcal{E}$).

That said, if the goal is maximizing predictive prowess then we must caution that FFtree’s faster generation of smaller models comes at a price- FFtrees usually generates significantly weaker predictions than DODGE($\mathcal{E}$) (see the RQ2 and RQ3 results, discussed later). Another reason to recommend DODGE($\mathcal{E}$) is that it generates better predictors than numerous recent SE state-of-the-art research articles [22, 3, 15, 2, 48, 24].

But more fundamentally, the other reason to explore DODGE($\mathcal{E}$) is that it tests the theory that much better hyperparameter optimizers can be built by assuming the output space divides into just a few regions of size $\mathcal{E}$. DODGE($\mathcal{E}$) is one way to exploit this effect. We believe that further research could be performed in many others ways (e.g., different learners, better visualizations and/or explanations of analytics, faster implementations of other tools).

1.1 Relation to Prior Work

All the DODGE($\mathcal{E}$) work is novel for this paper (this research team invented DODGE($\mathcal{E}$) and this is the first publication to discuss it). As to the work on FFtrees, for defect prediction, this paper includes the prior results with that of Chen et al. results [15]. The application of FFtrees to text mining (in this paper) is a novel result.

2 Background

This section describes the background on defect prediction, and text mining and the corresponding data and methods which are considered baselines.

2.1 Text Mining

Many SE project artifacts come in the form of unstructured text such as word processing files, slide presentations, comments, Github issue reports, etc. According to White [63], 80% of business is conducted on unstructured data, 85% of all data stored is held in an unstructured format and unstructured data doubles every three months. Nadkarni and Yezhkova [42] say that 1,600 Exabytes of data appears in unstructured sources and that each year, humans generate more unstructured artifacts than structured.

Lately, there have been much interest in SE text mining [34, 37, 48, 2, 65, 32] since it covers a much wider range of SE activities. Text mining is harder than other case studies (like defect prediction) due to presence of free form natural language which is semantically very complex and may not conform to any known grammar. In practice, text documents require tens of thousands of attributes (one for each word). For example, consider NASA’s software project and issue tracking systems (or PITS) [34, 37] that contain text discussing bugs and changes in source code. As shown in Table II, our text data contains tens to hundreds of thousands of words (even when reduced to unique words, there are still 10,000+ unique words).

2.1.1 Data and Algorithms for Text Mining

Table II describes our PITS data, which comes from six different NASA systems (which we label PitsA, PitsB,…etc). For this study, all datasets were preprocessed using the usual text mining filters [19]. We implemented stop words removal using NLTK toolkit [12] (to ignore very common short words such as “and” or “the”). Next, Porter’s stemming filter [52] was used to delete uninformative word endings (e.g., after performing stemming, all the following words would be rewritten to “connect”: “connection”, “connections”, “connective”, “connected”, “connecting”). After that, DODGE($\mathcal{E}$) selected other pre-processors using the space of options from Table I.

A standard text mining learner is SVM (support vector machine). A drawback with SVM is that its models may not be human comprehensible. Finding insights among unstructured text is difficult unless we can search, characterize, and classify the textual data in a meaningful way. One of the common techniques for finding related topics within unstructured text (an area called topic modeling) is the Latent Dirichlet allocation (LDA) [13]. LDA clusters text into “topics” defined by the high-frequency words in that cluster. For example, the topics found by LDA for one of our PITS data sets are shown in Table III. We studied LDA since it is a widely-used technique in prominent SE research articles [2].

LDA is controlled by various parameters (see Table I). At ICSE’13, Panichella et al. [48] used a genetic algorithm to tune their LDA text miners. More recently, in the IST’18 journal article, Agrawal et al. [2] saw that differential evolution can out-perform genetic algorithms for tuning LDA.

A standard pre-processor for text mining is vectorization; i.e., replace the raw observations of wordX appearing in documentY with some more informative statistic. For example, Agrawal et al. converted the PITS text data into the vectors of Table IV. The cells in that table shows how much each issue report matches each topic (and the final column shows the issue severity of that report). Table I lists the options for the LDA vectorization, plus three other vectorization methods.

2.2 Defect Prediction

Software developers are smart, but sometimes make mistakes. Hence, it is essential to test software before the deployment  [45, 8, 67, 41]. Software quality assurance budgets are finite but increasing assessment effectiveness by some linear amount can take exponentially more effort [22]. Therefore, standard practice is to apply the best available methods on code sections that seem most critical and bug-prone. Software bugs are not evenly distributed across the project [26, 30, 46, 39]. Hence, a useful way to perform software testing is to allocate most assessment budgets to the more defect-prone parts in software projects. Software defect predictors are never 100% correct. But they can be used to suggest where to focus more expensive methods.

There is much commercial interest in defect prediction. In a survey of 395 practitioners from 33 countries and five continents, Wan et al. [61] found that over 90% of the respondents were willing to adopt defect prediction techniques. When Misirli et al. [39] built a defect prediction model for a telecommunications company, those models could predict 87% of files with defects. Those models also decreased inspection efforts by 72%, and hence reduced post-release defects by 44%.

Software defect predictors not only save labor compared with traditional manual methods, but they are also competitive with certain automatic methods. A recent study at ICSE’14, Rahman et al.  [53] compared (a) static code analysis tools FindBugs, Jlint, and PMD and (b) static code defect predictors (which they called “statistical defect prediction”) built using logistic regression. They found no significant differences in the cost-effectiveness of these approaches.

Given this equivalence, it is significant to note that static code defect prediction can be quickly adapted to new languages by building lightweight parsers to extract static code metrics such as Table V. The same is not true for static code analyzers - these need extensive modification before they can be used in new languages.

2.2.1 Data and Algorithms for Defect Prediction

Our defect predictors where applied to the data described in Table VI. As shown in Table VI, this data is available for multiple software versions (from http://tiny.cc/seacraft). This is important since, an important principle of data mining is not to test on the data used in training. There are many ways to design a experiment that satisfies this principle. Some of the methods where we do not test data mining on training data itself have limitations too; e.g., leave-one-out is too slow for large data sets and cross-validation mixes up older and newer data  (such that data from the past may be used to test on future data). In this work, for each project data, we set the latest version of project data as the testing data and all the older data as the training data. For example, we use $\mathit{poi1.5,poi2.0,poi2.5}$ data for training predictors, and the newer data, $\mathit{poi3.0}$ is left for testing.

Table VI illustrates the variability of SE data. The data can be observed to have imbalanced class frequencies. If the target class is not common (as in the camel, ivy, etc test data in Table VI), it is difficult to generate a model that can locate it. A standard trick for class imbalance is SMOTE [14] that synthetically create members of the minority class. Table I show controlled parameters of SMOTE.

As to machine learning algorithms, there are many and varied. At ICSE’15, Ghotra et al. [24] applied 32 different machine learning algorithms to defect prediction. In a result consistent with the theme of this article, they found that those 32 algorithms formed into four groups of Table 9 in [24] (and the performance of two learners in any one group were statistically indistinguishable from each other).

2.3 Evaluation

2.3.1 Measures of Performance

We eschew precision and accuracy since these can be inaccurate for data sets where the target class is rare (which is common in defect prediction data sets) [35]. For example, consider a test data set with 20% defective examples. A learner could be 80% accurate for that data set, while still missing 100% of the defective examples. As to why we deprecate precision, we refer the interested reader to a prior work [35].

Instead, we will evaluate our predictors on metrics that aggregate multiple metrics. D2h, or “distance to heaven”, shows how close scores fall to “heaven” (where recall=1 and false alarms (FPR)=0) [15]. D2h was used to evaluate both defect predictors as well as text mining.

[TABLE]

Here, the $\sqrt{2}$ term normalizes d2h to the range zero to one.

For defect prediction, Popt(20) comments on the effort required after a defect predictor triggers and humans have to read code, looking for errors. Popt(20) is a specialized metric which can be used only with defect predictor. $Popt(20)=1-\Delta_{opt}$, where $\Delta_{opt}$ is the area between the effort (code-churn-based) cumulative lift charts of the optimal learner and the proposed learner. To calculate Popt(20), we divide all the code modules into those predicted to be defective ($D$) or not ($N$). Both sets are then sorted in ascending order of lines of code. The two sorted sets are then laid out across the x-axis, with $D$ before $N$. This layout means that the x-axis extends from 0 to 100% where lower values of $x$ are predicted to be more defective than $x$ higher values. On such a chart, the y-axis shows what percent of the defects would be recalled if we traverse the code sorted that x-axis order. Following from the recommendations of Ostrand et al. [46], Popt is reported at the 20% point; show how many bugs are find if we inspect a small portion of the code (20%).

Kamei, Yang et al.  [66, 28, 40] normalized Popt using:

[TABLE]

where $S(optimal)$, $S(m)$ and $S(worst)$ represent the area of curve under the optimal learner, proposed learner, and worst learner. Note that the worst model is built by sorting all the changes according to the actual defect density in ascending order. After normalization, Popt(20) (like d2h) has the range zero to one. Please note two important points. Firstly, unlike the defect prediction data of Table VI, the data for text mining task is not conveniently divided into versions. Hence, to generate separate train and test data sets, we use a $x*y$ cross-validation study where, $x=5$ times, we randomize the order of the data then divide into $y=5$ bins. Then, we test on that bin after training on all the others. Secondly:

• •

larger values of Popt(20) are better;

• •

smaller values of d2h are better.

2.3.2 Statistical Analysis

As to statistical methods, the following results use two approaches. Firstly, when comparing one result to a sample of $N$ others, we will sometime see “small effects” (which can be ignored). To define “small effect”, we use Cohen’s delta [16]:

[TABLE]

i.e., 20% of the standard deviation.

Secondly, other statistical tests are required when comparing results from two samples; e.g., when two variants of some stochastic process are applied, many times, to a population. For this second kind of comparison, we need a statistical significance test (to certify that the distributions are indeed different) and an effect size test (to check that the differences are more than a “small effect”). There are many ways to implement second kind of test. Here, we used those which have been past peer reviewed in the literature [3, 2]. Specifically, we use Efron’s 95% confidence bootstrap procedure [18] and the A12 test [6]. In this second test, to say that one sample $S_{1}$ is “worse” than another sample $S_{2}$ is to say: 1) The mean Popt(20) values of $S_{1}$ are less than $S_{2}$; 2) The mean D2h values of $S_{1}$ are more than $S_{2}$; and 3) The populations are not statistically similar; i.e., (a) their mean difference is larger than a small effect (using A12) and that (b) a statistical significance test (bootstrapping) has not rejected the hypothesis that they are different (at 95% confidence). Note we do not use A12 or bootstrap for the first kind of test, since those statistics are not defined for comparisons of individuals to a sample.

3 Motivation for new work: Surprising Results from FFtrees

This section describes the FFtrees results published by Chen et al. in FSE’18 [15] that (a) motivated this article and (b) lead to our hypothesis that “redundant parameter choices might be leading to indistinguishable results”. This will in turn lead to (next section) a new method called DODGE($\mathcal{E}$) that deprioritizes choices that lead to redundant results.

Fast and Frugal Trees (FFtrees) were developed by psychological scientists [33] trying to generate succinct, easily comprehensible models. FFtrees are binary trees that return a binary classification (e.g., true, false). Unlike standard decision trees, each level of an FFtree must have at least one leaf node. For example, Table VII shows an FFTtree generated from the log4j JAVA system of Table VI. The goal of this tree is to classify a software module as “defective=true” or “defective=false”. The four nodes in the Table VII FFTree reference four attributes cbo, rfc, dam, amc (defined in Table V).

Following the advice of [51, 15], we generate trees of depth of $d=4$. This means that FFtrees make their decisions using at most four attributes (where numeric ranges have been binarized by splitting at the median point).

Standard rule learners select ranges that best select for some goal (e.g., selecting for the “true” examples). This can lead to overfitting. To avoid overfitting, FFtrees use a somewhat unique strategy: at each level of the tree, FFtrees builds two trees using the ranges that most and least satisfy some goal; e.g., d2h or Popt20. That is, half the time, FFtrees will try to avoid the target class by building a leaf node that exits to “false”. Assuming a maximum tree depth of $d=4$ and two choices at each level, then FFtree builds $2^{d}=16$ trees then prunes away all but one, as follows:

• •

Firstly, select a goal predicate; e.g., d2h or Popt20.

• •

Next, while building one tree, at each level of the tree, FFtree scores each range according to how well that range {does, does not} satisfy that goal. These selected range becomes a leaf note. FFtree then calls itself recursively on all examples that do not fall into that range.

• •

Finally, while assessing 16 trees, the training data is run through each tree to find what examples are selected by that tree. Each tree is scored by passing the selected examples through the goal predicate.

• •

The tree with the best score is applied to the test data.

In summary, FFtrees explore around a few dozen times, trying different options for how to best model the data (i.e., what exit node to use at each level of the tree). After a few explorations, FFtrees deletes the worst models, and uses the remaining model on the test data.

Figure 3 shows results from Chen et al. [15] that compared FFtrees to standard defect predictors. In that comparison, Ghotra et al. [24] was used to guide learner selection. They found that 32 defect predictors group together into just four ranks from best to worse. (Please look for all four groups in Table 9 of [24]). We picked at random from each of their ranks to select SL=Simple Logistic, NB=Naive Bayes, EM=Expectation Maximization, SMO=Sequential Minimal Optimization (a kind of support vector machine). We call these learners “standard” since, in Figure 3, we use them with their defaults from Scikit-learn [49]. In Figure 3:

• •

Performance is evaluated using metrics from §2.3.

• •

Data comes from Table VI.

• •

This data has the attributes of Table V.

• •

For data with multiple versions, we test on the latest version and train on a combination of all the rest.

• •

If FFtrees perform worse than any other learner by more than a “small effect” (defined using Equation 5), then that result is highlighted in red (see the synapse d2h results of Figure 3). For each column, the size of a “small effect” is listed at top.

As shown in Figure 3, FFtrees nearly always performs as well, or better, than anything else.

4 Research Method: The DODGE($\mathcal{E}$) Algorithm

It is very surprising that something as simple as FFtree perform so well (see Figure 3), especially since:

• •

FFtrees explores very few alternate models (only 16).

• •

Each model references only four attributes.

• •

To handle numeric variables, a very basic discretization policy is applied at each level of tree (numerics are separated at the median value).

• •

Strange to say, half the time, FFtree’s overfitting mechanism will try to avoid the target class when it selects a leaf node that exits to “false”.

Under what conditions would something that simple work as well as the other methods shown in Figure 3? One possible answer was offered in the introduction. If the data has large $\mathcal{E}$ in its output space, then:

• •

The output/objective space has just a few cells; so

• •

If there are $c$ cells and $t$ tunings, and when $t>c$, then some of those will be redundant; i.e., they achieve results within $\mathcal{E}$ of other results.

• •

Which means that exploring around $c$ times will cover much of the output space.

If that is true, then to do better than FFtrees:

• •

Try exploring around across a wider range of options.

• •

If some options result in a performance score $\alpha$, then we will deprecate options that lead to $\alpha\pm\mathcal{E}$.

To find a wider range of options, DODGE($\mathcal{E}$) uses the Table I tree of options. Leaves in that tree are either:

• •

Single choices; e.g., DecisionTree, “splitter=random”; or

• •

Numeric ranges; e.g., Normalizer, “norm=l2”.

Each node in the tree is assigned a weight $w=0$. When evaluating a branch, the options in that branch configure, then executes, a pre-processor/learner. Each evaluation selects one leaf from the learner sub-tree and one from the pre-processing tree (and defect prediction and text mining explores different pre-processing sub-trees, see Table I). If the evaluation score is more than $\mathcal{E}$ of prior scores, then all nodes in that branch are endorsed ($w=w+1$). Otherwise, DODGE($\mathcal{E}$) deprecates ($w=w-1$). DODGE($\mathcal{E}$) uses these weights to select options via a recursive weighted descent where, at each level, it selects sub-trees whose root has the largest weight (i.e., those most endorsed).

The design conjecture of DODGE($\mathcal{E}$) is that exploring some tuning options matters but, given a large $\mathcal{E}$ output space, the details of those options are not so important. Hence, a limited number of $N_{1}$ times, we pick some options at random. Having selected those options, for further $N_{2}$ samples, we learn which of the $N_{1}$ options should be most deprecated or endorsed.

When a parameter range is initially evaluated, a random number $r=random(\mathit{lo},\mathit{hi})$ is selected and its weight $w(r)$ is set to zero. Subsequently, this weight is endorsed/deprecated technique as described above, with one refinement. When a new value is required (i.e., when the branch is evaluated again) then DODGE($\mathcal{E}$) restricts the $\{\mathit{lo},\mathit{hi}\}$ range as follows. If the best,worst weights seen so far (in this range) are associated with $b,w$ (respectively) then use $\mathit{lo}=b$ and $\mathit{hi}=\frac{b+w}{2}$. Important point: endorse and deprecate is done each time a branch is evaluated within each $N_{1}$ and $N_{2}$ steps. Figure 4 provides a summarized procedure on how DODGE($\mathcal{E}$) works.

In summary, DODGE($\mathcal{E}$) is a method for learning what tunings are redundant; i.e., lead to results that are very similar to other tunings [1]. It is controlled by two meta-parameters:

• •

$\mathcal{E}$: results are “similar” if they differ by less than $\mathcal{E}$;

• •

$N$: the number of sampled tunings.

Recall that $N=N_{1}+N_{2}$ where

• •

The first $N_{1}$ times, the set of tuning options grows;

• •

For the remaining $N_{2}$ times, that set is frozen while we refine our understanding of what tunings to avoid.

5 Experimental Results

Using DODGE($\mathcal{E}$), we can now answer the research questions asked in this article’s introduction.

5.1 RQ1: Is DODGE($\mathcal{E}$) too complicated? How to find appropriate value of $\mathcal{E}$?

Firstly, we wanted to verify whether our hypothesis of ”redundant options (similar region defined within $\mathcal{E}$) might be leading to indistinguishable results”. To test this, we use Figure 5 to see how quickly (i.e., after how many evaluations $N$) the performance of DODGE($\mathcal{E}$) plateaus. In figure 5, the X-axis represents number of samples ($N$) and the Y-axis represents the max value of $P_{opt}$ seen until that sample (and for that measure, larger values are better). We used $\mathcal{E}$ of 0.05, 0.1, and 0.2 values, and looked for the number of samples needed before the performance plateaus. We also show the performance variability measured in terms of the interquartile range (IQR) (which is the (75-25)th percentile). These IQR values are very small; i.e., DODGE($\mathcal{E}$)’s performance is very stable.

In Figure 5, we observe that most change in improvement happens after just few tens of evaluations. This supports our hypothesis that there are ”redundant options which lead to indistinguishable results”. Figure 6 and Figure 7 explore different settings of $\{N,\mathcal{E}\}$.

• •

Figure 6 varies $\mathcal{E}$ but keeps $N$ constant. In this treatment, we check how much improvement do we miss on when we try to find the right $\mathcal{E}$ value. Let’s say we cut the Figure 5 at $N=30$ line on x-axis, we report the values of $\mathcal{E}$ for 0.05, 0.1 and 0.2.

• •

Figure 7 varies $N$ but keeps $\mathcal{E}$ constant. Please note, when we say $\mathcal{E}$ constant, we wanted to see how much improvement do we miss on when we try to find the right $N$ value. Let’s say we look at $\mathcal{E}=0.2$ line, we report the values of $N$ for 30, 100 and 1000.

As shown in these figures, changes to $\{N,\mathcal{E}\}$ alter the performance of DODGE($\mathcal{E}$) by less than a “small effect”.

That is, (a) the output space for this data falls into a very small number of regions so (b) a large number of samples across a fine-grained division of the output space performs just as well as a few samples over a coarse-grained division.

In summary, our answer to RQ1 is that the values of $\{N,\mathcal{E}\}$ can be set very easily. Based on the results of Figure 6 and Figure 7, for the rest of this article we will use $\mathcal{E}=0.2$ while taking $N=30$ samples of the options from Table I. We observed that there is no significant loss in performance if we move $\mathcal{E}$ to different values or $N$ provided in figures 6 and 7. We picked $\mathcal{E}=0.2$ as we are looking for larger redundant region in data at the same time faster evaluations of $N=30$.

5.2 RQ2: How does DODGE($\mathcal{E}$) compare to recent prominent defect prediction and hyperparamter optimization results?

SMOTUNED is Agrawal et al. ICSE’18 [3]’s hyperparamater optimizer that tunes SMOTE, a data pre-processor (recall that SMOTE is a tool for addressing class imbalance and was described in §2.2.1). Agrawal et al. reported that SMOTUNED’s tunings greatly improved classifier performance. SMOTUNED uses differential evolutionary algorithm [55] and tunes the control parameters of SMOTE (see Table I).

DE+RF is a hyperparameter optimizer proposed by Fu et al. [23] that uses differential evolution to tune the control parameters of random forests. The premise of RF (which is short for random forests) is “if one tree is useful, why not a hundred?”. RF quickly builds many trees, each time using a random selection of the attributes and examples. The final conclusion is then generated by polling across all the trees in the forest. RF’s control parameters are listed in Table I.

SMOTUNED and DE+RF used DE since (a) DE can handle numeric and discrete options; and (b) it has proven useful in prior SE studies [22]. Further, other evolutionary algorithms (genetic algorithms [25], simulated annealing [29]) mutate each attribute in isolation. When two attributes are correlated, those algorithms can mutate variables inappropriately in different directions. DE, on the other hand, mutates attributes in tandem along known data trends. Hence, DE’s tandem search can outperform other optimizers such as (a) particle swarm optimization [60]; (b) the grid search used by Tantithamthavorn et al. to tune their defect predictors [57, 56]; or (c) the genetic algorithm used by Panichella et al.  [48] to tune a text miner (see below).

Figure 8 compares hyperparameter optimizers with DODGE(.2), FFtrees and (just for completeness) a random search method that picks 30 random options (equivalent N as of DODGE($\mathcal{E}$)) from Table I. These experiments make extensive use of stochastic algorithms whose behavior can significantly differ between each run (DE and Random30). Hence, Figure 8 shows mean results from 25 runs using 25 different seeds. In those results:

• •

Usually, random performs badly and never defeats DODGE($\mathcal{E}$). This result tells us that the reweighing scheme within DODGE($\mathcal{E}$) is useful.

• •

In 16/20 cases combining the d2h and Popt20 datasets, DODGE(.2) is no worse than anything else (where “worse” is defined as per §2.3.2).

• •

In two cases, DODGE(.2) is beaten by FFtrees (see the d2h results for jedit and log4j). That is, in 90% of these results, methods that explore a little around the results space do no worse than methods that try to extensively explore the space of tuning options.

In summary, our answer to RQ2 is that DODGE($\mathcal{E}$) often performs much better than recent prominent standard hyperparameter optimization results.

5.3 RQ3: Is DODGE($\mathcal{E}$) only useful for defect prediction?

DODGE($\mathcal{E}$) was designed in the context of defect prediction. This section checks if that design applies to a very different software analytics; i.e., SE text mining. Note that, as with defect prediction, hyperparameter optimizers (like DODGE($\mathcal{E}$)) adjust the control parameters of Table 1. In the particular case of text mining, we adjust the Table 1 text mining data pre-processing options (used to generate data sets like Table IV). We also adjust the Table 1 learner options.

Figure 9 shows our text mining results. As before, for completeness sake, we include results by RANDOMly selecting tuning and learning options.

As seen in Figure 9, in only one case DODGE($\mathcal{E}$)’s performance is worse than anything else (where “worse” is defined as per §2.3.2). The LDA-FFT results for PitsF is 2% better than DODGE($\mathcal{E}$), but difference was deemed insignificant by our statistical tests. And, just as with the Figure 8 results, when DODGE($\mathcal{E}$) fails, it is beaten by a treatment that uses FFtree (see the PitsB LDA-FFT results). That is, in 100% of these results, methods that explore a little around the results space do no worse than methods that try to extensively explore the space of tuning options (e.g., genetic algorithms and differential evolution).

In summary, our answer to RQ3 is that DODGE($\mathcal{E}$) is not just a defect prediction method. Its success with text mining make it an interesting candidate for further experimentation with other SE tasks.

6 Threats to Validity

This paper is not about how to generate good predictors, per se. Instead, it is more about an instrument (DODGE($\mathcal{E}$)) that probes the nature of the space options associated with AI tools applied to SE. We show that when prior work has tried to generate good predictors, their algorithms have been much slower than necessary since they waste much time exploring a large number of redundant options.

Nevertheless, our experimental rig repeats numerous prior studies (this time adding in DODGE($\mathcal{E}$)). Such is the nature of repeated studies that our work shares the same threats to validity as that of prior work (discussed below).

Sampling Bias: This article shares the same sampling bias problem as every other data mining paper. Sampling bias threatens any classification experiment (since what matters in one data set may or may not hold in another). For example, one of our sampling biases is that all our data comes from open source projects.

Having said that, our sample bias is somewhat smaller than other papers since, we applied our frameworks to 16 SE data sets giving us more conclusive results. Also, we showed that there exists more than one domain where DODGE($\mathcal{E}$) is a useful approach.

Learner Bias: For building different classifiers in this study, we used many preprocessors (13) and learners (6). We chose these learners because past studies have shown that, these have been extensively used [24, 3, 57, 56]. Thus they are selected as the state-of-the-art learners to be compared with DODGE($\mathcal{E}$). In theory, there exists other learners (which we have not explored) and could change our results.

One important class of learners not studied here are those that use numerous hyperparameters. All the learners explored here by DODGE have less than dozen hyperparameters. In the future, it would be worth studying the value of DODGE on more complex machine learning algorithms such as neural networks.

Evaluation Bias: This paper uses two performance measures, i.e., $P_{opt}$ and $dist2heaven$. Other quality measures are often used in software engineering to quantify the effectiveness of prediction  [35, 38, 27]. We used these measures since we wanted to show the success of DODGE($\mathcal{E}$) for multi-goals and these two measures are more prominent in the literature.

Order Bias: For the performance evaluation part, the order that the data trained and predicted can affect the results. Also, for the defect prediction datasets, we deliberately choose an ordering that mimics how our software projects releases versions so, for those experiments, we would say that bias was required and needed. Further, for the other text mining datasets, to mitigate this order bias, we ran our rig in a 5-bin cross validation 5 times, randomly changing the order of the data each time.

Construct Validity: At various stages of data collection by different researchers, they must have made engineering decisions about what object-oriented metrics need to be extracted. Though all these decisions have been verified and evaluated by past researchers [3, 2] to make sure the dataset collection do not suffer from any construct validity.

External Validity: DODGE($\mathcal{E}$) self-selects the tunings used in the pre-processors and data miners. Hence, by its very nature, this article avoids one threat to external validity (i.e., important control parameter settings are explored).

This paper reports results from two tasks (defect prediction and text mining) to show that the same effect holds in both tasks; i.e., algorithms can be remarkably effective when they assume the output space seems to divide into a very small number of regions. Most software analytics papers report results from one task; i.e., either defect prediction or text mining. In that sense, the external validity of this paper is greater than most analytics papers.

On the other hand, this paper only reports results from two tasks. There are many more kinds of SE tasks that should be explored before it can be conclusively stated that DODGE($\mathcal{E}$) is widely applicable and useful.

Another threat to external validity is that this article compares DODGE($\mathcal{E}$) against existing hyperparameter optimization in the software analytics literature. We do not compare our new approach against the kinds of optimizers we might find in search-based SE literature [50]. There are two reasons for this. Firstly, search-based SE methods are typically CPU intensive and so do not address our faster termination goal. Secondly, the main point of this article is to document a previous unobserved feature of the output space of software analytics. In order to motivate the community to explore that space, some article must demonstrate its existence and offer an initial results showing that, using the knowledge of output space, it is possible to do better than past work.

7 Related Work

DODGE($\mathcal{E}$) is a novel hyperparameter optimizer. This section offers some brief notes on other research into hyperparameter optimizers. Note that applications of hyperparameter optimization to software engineering is a very large topic. Elsewhere [4] we offer an extensive literature review on hyperparameter optimization and its applications in software engineering. Here, we offer some overview notes.

Apart from DODGE($\mathcal{E}$), there are many ways to implement hyperparameter optimizers. For example, grid search [9] creates $C$ nested for-loops to explore $C$ control parameters. Bergstra et al. deprecate grid search arguing that (a) the best hyperparameters are usually found within a very small region of the total space; and (b) a grid search that is fine-grained enough to find that region for any learner and any data set would be very slow indeed [9]. Despite this, some SE researchers persist in using grid search [57, 56].

Another way to implement hyperparamter optimization is random search [9]. This approach sets up ranges of hyperparameter values and select random combinations to train the model and evaluate. There are many other ways to implement this kind of optimization including those that use some form of genetic algorithm like differential evolution [55], NSGA-2 [17], IBEA [70], or MOEA/D [68]. In this paper, we have already seen examples of the these standard hyperparameter optimizers. For example, LDA-GA SVM used its own genetic algorithm while LDADE FFT and LDADE SVM both used differential evolution [55]. As shown in Figure 2, those algorithms took (much) longer to execute and (measured in terms of d2h, usually perform worse than as DODGE($\mathcal{E}$)).

The slowness of standard hyperparameter optimizers restricts the space of hyperparameters that can be explored. For example, Arcuri & Fraser [7] warn that “the possible number of parameter combinations is exponential in the number of parameters, so only a limited number of parameters and values could be used.”. We conjecture that if they used DODGE($\mathcal{E}$), then they could have explored more parameters and possibly reversed their conclusion that hyperparameter optimization adds little extra value. Since Arcuri & Fraser’s 2013 study, other researchers in that research sub-area (test case generation) have found hyperparameter tuning very useful; e.g., see the 2015 study by Panichella et al. [47].

The Panichella et al. study is very relevant to this paper since their 12,800 experiments (each with a give-up time of 600 seconds), required 12.7 weeks of CPU to terminate. We conjecture that with tools like DODGE($\mathcal{E}$), more studies like Panichella et al. could be completed, much quicker, with far fewer resources.

More generally, DODGE($\mathcal{E}$) could speed up standard hyperparameter optimization. We conjecture that those optimizers could run much faster if they pruned away redundant evaluations using DODGE($\mathcal{E}$). If that were true then DODGE($\mathcal{E}$) could have a very large impact over a very wide range of research.

There is another way that DODGE($\mathcal{E}$) comments on standard optimization methods. Landscape analysis is the process of exploring a large complex problem/solution space in order to learn its shape. Once that is learned, then different search strategies could be proposed to better survey that particular shape. One drawback with landscape analysis is that it can be extremely computationally expensive. To learn the landscape associated with the test suites of 19 software programs, 30 times, Aleti et al. [5] evaluated 1,000,000 test suites. The lesson of DODGE($\mathcal{E}$) is that, sometimes, landscapes can be mapped without requiring 19301,000,000=570 million evaluations. For example, in this paper, we assumed a particular “landscape” (see Figure 1) then designed a search method, DODGE($\mathcal{E}$), that would succeed quickly if that landscape existed, or fail badly otherwise. Perhaps this strategy could be used in future research to reduce the cost of landscape analysis.

8 Conclusion

This article has discussed ways to reduce the CPU cost associated with hyperparameter optimization for software analytics. Tools like FFtrees or DODGE($\mathcal{E}$) were shown to work as well, or better, than numerous recent SE results:

• •

FFtrees work so well since the output space looks like Figure 1 (i.e., it contains only a few regions where results can be distinguished from each other). In that space, FFtree’s limited probing serves to sample the space.

• •

DODGE($\mathcal{E}$) works better than FFtrees since the deprecation strategy of Figure 4 is a better way to sample Figure 1 than FFtree’s random probes.

• •

Other methods (used in prior SE research) perform worse than DODGE($\mathcal{E}$) since they do not appreciate the simplicity of the output space (where “simplicity” means that it only contains a few distinct results). Hence, those other methods waste much CPU as they struggle to cover billions of tuning options like Table I (most of which yield indistinguishably different results).

Generalizing from our results, perhaps it is time for a new characterization of software analytics:

*Software analytics is that branch of machine learning that studies problems with large $\mathcal{E}$ outputs. *

This new characterization is interesting since it means that a machine learning algorithm developed in the AI community might not apply to SE. A similar conclusion has recently been offered by Binkley et al. who argue for SE-specific information retrieval methods [11].

We suspect that understanding SE is a different problem to understanding other problems that are more precisely retrained. Perhaps, it is time to design new machine learning algorithms (like DODGE($\mathcal{E}$)) that are better suited to large $\mathcal{E}$ SE problems. As shown in this article, such new algorithms can exploit the peculiarities of SE data to dramatically improve software analytics.

We hope that this article inspires much future work on a next generation of SE data miners. For example, tools like DODGE($\mathcal{E}$) need to be applied to more SE tasks to check the external validity of these results. Another useful extension to this work would be to explore problems with three or more goals (e.g., reduce false alarms while at the same time improving precision and recall). Further, as discussed in the Related Work section, there are research opportunities where (a) DODGE($\mathcal{E}$) is used to repeat and improve prior work or (b) speed up a wide range of other search-based SE algorithms (by using redundancy pruning to reduce the space of candidate mutations).

Lastly, there are many ways in which DODGE($\mathcal{E}$) could be improved. Right now we only deprecate tunings that lead to similar results. Another approach would be to deprecate tunings that lead to similar and worse results (perhaps to rule out parts of the output space, sooner). Also, it would be useful if the Table I list could be reduced to a smaller, faster to run, set of learners. That is, here we could select learners which can terminate faster while generating the most variable kinds of models.

Acknowledgements

This work was partially funded by an NSF Grant #1703487.