Model-based Automated Testing of JavaScript Web Applications via Longer Test Sequences

TL;DR

This paper introduces LJS, a model-based testing tool for JavaScript web apps that uses longer event sequences to improve code coverage efficiently, outperforming existing tools like JSDep.

Contribution

It presents a novel approach leveraging longer test sequences for higher coverage and implements a tool that demonstrates significant improvements over prior methods.

Findings

LJS achieves 86.4% line coverage in 10 minutes.

LJS outperforms JSDep by 5.4% on average.

Longer test sequences lead to higher code coverage.

Abstract

JavaScript has become one of the most widely used languages for Web development. However, it is challenging to ensure the correctness and reliability of Web applications written in JavaScript, due to their dynamic and event-driven features. A variety of dynamic analysis techniques for JavaScript Web applications have been proposed, but they are limited in either coverage or scalability. In this paper, we propose a model-based automated approach to achieve high code coverage in a reasonable amount of time via testing with longer event sequences. We implement our approach as the tool LJS, and perform extensive experiments on 21 publicly available benchmarks (18,559 lines of code in total). On average, LJS achieves 86.4\% line coverage in 10 minutes, which is 5.4\% higher than that of JSDep, a breadth-first search based automated testing tool enriched with partial order reduction. In particular, on large applications, the coverage of LJS is 11-18\% higher than that of JSDep. Our empirical finding supports that longer test sequences can achieve higher code coverage in JavsScript testing.

Tables

Table 1. TABLE I: Coverage of LJS and JSDep in 600s.

Name	Loc	LJS				JSDep
		CRG.	sd	Tests	Len.	CRG.	Tests	M.Len.
case1	59	100.0%	0	4679	99	100.0%	1409	705
case2	72	100.0%	0	4376	99	100.0%	3058	549
case3	165	100.0%	0	2739	99	100.0%	7811	575
case4	196	87.0%	0	2816	99	77.9%	8594	500
frog	567	96.8%	0.004	15	99	84.6%	86	16
cosmos	363	82.0%	0.060	322	99	79.5%	973	243
hanoi	246	89.0%	0	1303	99	82.5%	902	225
flipflop	525	97.0%	0	59	99	96.3%	284	71
sokoban	3056	88.6%	0.026	58	99	77.6%	203	51
wormy	570	45.4%	0.054	35	99	41.0%	323	18
chinabox	338	84.0%	0	7	99	82.3%	92	9
3dmodel	5414	85.0%	0	8	99	71.5%	66	10
cubuild	1014	88.8%	0.060	7	99	72.8%	153	17
pearlski	960	55.0%	0	72	99	54.9%	214	52
speedyeater	784	89.8%	0.010	516	99	82.1%	1497	374
gallony	300	95.0%	0	2133	99	94.5%	1611	95
fullhouse	528	92.2%	0.012	1119	99	86.3%	889	222
ball_ool	1745	92.8%	0.004	1	99	74.2%	18	4
harehound	468	94.8%	0.004	522	99	94.5%	1224	116
match	369	72.8%	0.004	1063	99	73.2%	4050	845
lady	820	79.0%	0	0	99	75.7%	35	8
Average	883.8	86.4%	0.011	1040.5	99	81.0%	1594.9	-

Table 2. TABLE II: Comparison of state abstraction techniques.

Name	State abstraction from [2]				Our state abstraction
	CRG.	$\left|S\right|$	$\left|\delta \right|$	Time(s)	CRG.	$\left|S\right|$	$\left|\delta \right|$	Time(s)
case1	100.0%	1.0	2.0	0.5	100.0%	1.0	2.0	0.0
case2	97.6%	1.0	4.0	0.7	95.2%	1.0	4.0	0.9
case3	100.0%	1.0	6.0	1.2	99.0%	1.0	6.0	1.4
case4	87.0%	1.0	8.0	1.3	87.0%	1.0	8.0	1.5
frog	95.2%	199.0	198.0	87.5	94.8%	24.2	100.8	105.9
cosmos	79.0%	125.0	196.0	9.0	78.8%	65.6	142.2	10.1
hanoi	89.0%	104.8	186.0	2.0	89.0%	99.0	185.4	2.3
flipflop	97.4%	25.8	100.4	21.6	97.0%	27.2	110.8	18.5
sokoban	88.4%	68.6	191.4	23.6	88.6%	31.2	125.8	26.4
wormy	42.2%	189.8	198.0	37.9	40.8%	132.8	185.4	38.6
chinabox	84.0%	67.6	156.4	136.8	84.0%	69.8	161.4	151.2
3dmodel	81.6%	3.0	25.6	105.5	83.2%	1.0	6.0	106.2
cubuild	85.0%	87.0	166.6	131.8	84.8%	87.4	163.2	132.5
pearlski	55.0%	136.8	196.4	18.2	55.0%	75.6	176.8	19.2
speedyeater	82.0%	175.8	198.0	6.0	81.2%	4.6	65.0	9.5
gallony	95.0%	63.8	174.8	1.4	95.0%	63.4	176.0	1.7
fullhouse	93.0%	161.6	198.0	2.5	93.0%	28.6	116.0	3.3
ball_pool	93.0%	40.6	142.2	354.4	93.0%	39.8	150.4	346.2
harehound	83.8%	191.0	198.0	5.9	84.8%	16.2	109.4	9.0
match	67.4%	13.6	171.0	3.2	69.8%	4.4	62.8	5.0
lady	79.2%	173.4	197.8	936.1	79.2%	85.6	174.0	1109.6
Average	84.5%	87.2	138.8	89.9	84.4%	41.0	106.3	100.0

Table 3. TABLE III: Comparison of event selection strategies.

Name	Random Event Selection				Weighted Event Selection
	CRG.	$\left|S\right|$	$\left|\delta \right|$	Time(s)	CRG.	$\left|S\right|$	$\left|\delta \right|$	Time(s)
case1	100.0%	1.0	2.0	0.5	100.0%	1.0	2.0	0.6
case2	100.0%	1.0	4.0	0.9	97.6%	1.0	4.0	1.4
case3	98.0%	1.0	6.0	1.9	100.0%	1.0	6.0	2.2
case4	87.0%	1.0	8.0	1.4	87.0%	1.0	8.0	1.7
frog	95.2%	28.6	101.0	107.1	95.2%	52.0	119.8	208.9
cosmos	79.0%	66.6	143.6	9.5	76.8%	24.0	149.0	9.0
hanoi	89.0%	102.8	186.4	2.1	89.0%	112.0	121.4	3.5
flipflop	97.0%	26.6	108.2	20.3	97.0%	17.0	55.8	50.6
sokoban	89.6%	28.6	123.2	24.8	84.0%	9.0	36.0	25.4
wormy	40.6%	131.8	186.8	37.1	41.8%	134.0	174.0	37.8
chinabox	84.0%	60.0	151.6	159.2	84.0%	72.2	159.8	129.4
3dmodel	82.0%	1.0	6.0	98.1	84.0%	1.0	6.0	89.1
cubuild	84.8%	92.8	171.2	136.9	85.6%	85.4	141.4	130.1
pearlski	55.0%	71.2	175.0	18.9	51.0%	33.0	87.0	17.5
speedyeater	87.6%	4.8	65.2	8.9	82.2%	2.0	34.0	9.9
gallony	95.0%	61.6	173.8	1.5	92.0%	30.4	127.6	2.1
fullhouse	93.0%	30.6	123.0	3.1	75.0%	5.0	13.0	4.7
ball_pool	93.4%	42.2	146.2	345.1	92.8%	54.6	191.0	404.9
harehound	89.4%	11.8	99.2	9.1	88.2%	3.0	34.0	10.4
match	69.4%	3.6	56.4	4.8	69.6%	4.0	50.6	5.6
lady	79.0%	82.6	171.4	1849.2	78.0%	67.4	153.6	1881.1
Average	85.1%	40.5	105.2	135.3	83.4%	33.8	79.7	144.1

Table 4. TABLE IV: Comparison of event sequence generation algorithms, where coverage (i.e. CRG) only includes that of event sequence execution.

Name	Baseline with POR				Long Sequence Generation
	CRG.	Len.	Time(s)	Tests	CRG.	Len.	Times	Tests
case1	100.0%	16	829	65504	100.0%	99	600	4543
case2	87.8%	9	1125	87040	100.0%	99	600	4244
case3	68.6%	7	931	54432	100.0%	99	600	2658
case4	61.8%	6	674	32768	87.0%	99	600	2732
frog	88.6%	4	664	592	95.8%	99	600	8
cosmos	78.0%	5	1072	4415	83.8%	99	600	175
hanoi	86.8%	6	889	4276	89.0%	99	600	1365
flipflop	97.0%	6	1200	419	97.0%	99	600	66
sokoban	88.6%	5	1200	1697	89.0%	99	600	54
wormy	41.2%	13	657	317	43.2%	99	600	30
chinabox	78.0%	8	729	64	84.0%	99	600	5
3dmodel	72.0%	4	1200	93	85.0%	99	600	10
cubuild	75.2%	6	1019	238	92.2%	99	600	10
pearlski	52.1%	7	960	653	55.0%	99	600	85
speedyeater	82.3%	5	1200	17106	88.0%	99	600	508
gallony	94.5%	8	1200	9821	95.0%	99	600	2852
fullhouse	79.2%	8	1200	12097	79.0%	99	600	1261
ball_pool	92.1%	6	1200	22	93.0%	99	600	3
harehound	92.2%	4	1200	7949	95.0%	99	600	420
match	73.2%	5	738	13341	73.0%	99	600	1103
lady	73.2%	5	696	12	75.0%	99	600	6
Average	79.2%	-	980	14897.9	85.7%	99	600	1054.2

Full text

I Introduction

JavaScript is a highly dynamic programming language with first-class functions and “no crash” philosophy, which allows developers to write code without type annotations, and to generate and load code at runtime. Partially due to these programming flexibilities, Web applications based on JavaScript are gaining increasing popularity. These features are, however, double-edged sword, making these Web applications prone to errors and intractable to static analysis.

Dynamic analysis has proven to be an effective way to test JavaScript Web applications  [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21]. Since it requires testcases to explore the state space of the application, various approaches for automated testcase generation have been developed in literature, which can generate event sequences and/or associated input data of events. The event sequences concern the order in which event handlers are executed (e.g., the order of clicking buttons), while the input data concerns the choice of values (e.g., strings, numbers and objects). The generation of both event sequences and input data is important to achieve a high code coverage, and has been extensively studied.

In general, event sequences are generated by randomly selecting event handlers with heuristic search strategies [1, 2, 3, 10, 13]. These approaches are able to analyze large real-life applications, but are usually left with a low code coverage. One possible reason, as mentioned before [12, 22], is that the event sequences are insufficiently long to explore parts of the code which may trigger the error. For example, in an experiment of [12], the uncovered code of the benchmark tetris is mainly due to the function gameOver which will only be invoked after a long event sequence. For traditional white-box testing and GUI testing, it has been shown that increasing the length of test cases could improve coverage and failure detection [23, 24, 25, 26, 27]. However, this has not been fully exploited in testing JavaScript Web applications. One of the reasons is that existing approaches usually generate event sequences from scratch by iteratively appending events to the constructed sequences up to a maximum bound, and the number of event sequences may blow up exponentially in terms of this bound. Therefore, for efficiency consideration, the maximum bound often have to be small (for instance, less than 6 [12]). To mitigate these issues, pruning techniques (e.g. mutation testing [7, 15] and partial order reduction [22]) were proposed to remove redundant event sequence, which allow to explore limited longer event sequences in a reasonable time. On the other hand, the input data is generated by either randomly choosing values with lightweight heuristic strategies [2, 3, 5], or using heavyweight techniques (e.g., symbolic/concolic execution) [1, 6, 11, 12, 16, 17, 19]. These works either consider unit testing or usually simply reuse the aforementioned methods to generate event sequences.

In this work, we focus on the event sequence generation. In particular, we propose a novel model-based automated testing approach to achieve a high code coverage in a reasonable time by generating and executing long event sequences. Our approach mainly consists of two key components: model constructor and event sequence generator. The model constructor iteratively queries an execution engine to generate a finite-state machine (FSM) model. It explores the state space using long event sequences and in a way to avoid prefix event subsequences re-executing and backtracking. To improve scalability, we propose a state abstraction approach, as well as a weighted event selection strategy, to construct small-sized FSM models. The event sequence generator creates long event sequences by randomly traversing the FSM model from the initial state. We implement our approach in a tool Longer JavaScript (LJS). To compare with other methods, we also implemented a random event selection strategy, and event sequence generator from JSDep [22] based on the FSM model.

In summary, the contributions of this paper are

• •

A new model-based automated testing approach for client-side JavaScript Web applications via longer event sequences;

• •

An implementation of our approach as a tool LJS; and

• •

An evaluation of 21 benchmarks from JSDep [22] including 18,559 lines of code in total.

LJS is available from https://github.com/JavaScriptTesting/LJS. On average, it achieves 86.4% line coverage in 10 minutes and is 5.4% higher than that of the tool JSDep [22]. On large applications, the coverage of LJS is 11-18% higher than that of JSDep. We have found that long event sequences can indeed improve the coverage with respect to the application under test. This paper then provides concrete, empirically validated approaches to generate long event sequences.

Structure. Section II introduces preliminaries, a running example and an overview of LJS. Section III describes the methodology of LJS. Section IV presents experimental results. Section V discusses related work. We conclude in Section VI.

II Preliminaries and Overview

II-A Preliminaries

II-A1 JavaScript Web Application

A client-side Web application, which is the main focus of the paper, consists of HTML/Script files which are executed by a Web browser. When a browser loads a Web page, it parses the HTML/Script files, represents them as a Document Object Model (DOM) tree, and then executes the top-level script code. Each node in the DOM tree represents an object on the Web page and may also be associated with a set of events. Each event may have some event handlers (e.g., callback functions such as onload and onclick) which are either statically registered inside the HTML file or dynamically registered by executing functions. When an event occurs (e.g., a button is clicked), the corresponding event handlers are executed sequentially. Although the browser ensures that each callback function is executed atomically, the execution of the entire Web application exhibits nondeterminism due to the interleaving of executions of multiple callback functions.

II-A2 DOM Event Dependency

Given a JavaScript Web application, let $R_{c}$ and $R_{d}$ respectively denote the control and data dependency relation over functions of the application. Formally, for each pair of functions (usually event handlers) $m_{1}$ and $m_{2}$, $(m_{1},m_{2})\in R_{c}$ (resp. $(m_{1},m_{2})\in R_{d}$) if there are two statements $st_{1}$ in $m_{1}$ and $st_{2}$ in $m_{2}$ that $st_{1}$ affects the control (resp. data) flow of $st_{2}$. Given two DOM events $e_{1}$ and $e_{2}$, $e_{2}$ is dependent on $e_{1}$, denoted by $e_{1}\rightarrow e_{2}$, if (1) there are event handlers $m_{1}$ and $m_{2}$ of $e_{1}$ and $e_{2}$ respectively such that $(m_{1},m_{2})\in(R_{c}\cup R_{d})^{*}$, or (2) the execution of $m_{1}$ registers, removes, or modifies $m_{2}$. Two event sequences $\rho_{1}$ and $\rho_{2}$ are equivalent if $\rho_{1}$ can be transformed from $\rho_{2}$ by repeatedly swapping adjacent and independent events of $\rho_{2}$.

II-A3 Finite State Machine

A (nondeterministic) Finite State Machine (FSM) is a tuple $M=(S,I,\delta,s_{0})$, where $S$ is a finite set of states with $s_{0}\in S$ as the initial state, $I$ is a finite input alphabet, $\delta\subseteq S\times I\times S$ is the transition relation. A transition $(s_{1},e,s_{2})\in\delta$ denotes that after reading the input symbol $e$ at the state $s_{1}$, the FSM $M$ can move from the state $s_{1}$ to the state $s_{2}$. We denote by ${\sf supp}(s)$ the set $\{(e,s^{\prime})\in I\times S\mid(s,e,s^{\prime})\in\delta\}$. Given a word $e_{1}\cdots e_{n}\in I^{*}$, a run of $M$ on $e_{1}\cdots e_{n}$ is a sequence of states $s_{0}s_{1}\cdots s_{n}$ such that for every $1\leq i\leq n$, $(s_{i-1},e_{i},s_{i})\in\delta$. We denote by $\epsilon$ the empty word. Note that, in this work, we will use an FSM to represent the behaviors of a JavaScript Web application. Because of this, we do not introduce the final states as in classic FSMs.

II-B Running Example

Consider the running example in Figure 1, where the HTML code defines the DOM elements of three checkboxes and one button. The JavaScript code defines a global variable count and five functions manipulating count and the DOM elements. Initially, the three checkboxes, named A-C, are unchecked; the button, named Submit, does not have any onclick event handler. The onclick event handlers of the three checkboxes are the functions FA-FC respectively. For each X$\in${A,B,C}, when the checkbox X is clicked (i.e., the corresponding event occurs), its state (checked/unchecked) is switched, and then the onclick event handler, namely the function FX, is executed. FX first determines whether the state of X is checked or not. If it is checked, then the global variable count increases by one, otherwise count decreases by one. Finally, FX invokes the function CheckedEnough to verify whether the value of count is no less than $3$, i.e., there are at least three checkboxes in the checked state. If count$>=$3, then the FSubmit function is registered to the onclick event handler of the button Submit; otherwise (i.e., count$<$3), the onclick event handler of the button Submit is removed. If the button Submit is clicked when FSubmit serves the onclick event handler of Submit, FSubmit gets executed and prints a message to the console.

In this example, for X,Y$\in${A,B,C}, the onclick event of X is dependent upon the onclick event of Y, and the onclick event of Submit is dependent upon the onclick event of X.

II-C Limitations of Existing Approaches

We now demonstrate why long event sequences are important for automated testing of JavaScript Web applications. Tools like Artemis [2] and JSDep [22] generate event sequences by systematically triggering various DOM events up to a fixed depth. After loading the Web page, these tools start by exploring all the available events at an initial state. If a new state is reached by executing a sequence of events, then all the available events at the new state are appended to the end of the event sequence. The procedure is repeated until time out or a fixed depth is reached. The search tree of our running example up to depth four is depicted in Figure 2, where the unshown labels of edges in black are respectively A,B,C, and those in red are Submit. Each edge labeled by X$\in${A,B,C,Submit} denotes the execution of the onclick event handler of X, and each node denotes a state. For each event sequence $\rho$ of length three, if $\rho$ is a permutation of A;B;C, then there are four available events A,B,C,Submit after $\rho$, otherwise there are three available events A,B,C.

A naïve algorithm (like the default algorithm in Artemis) would inefficiently explore the event space, i.e., the full tree in Figure 2, and may generate $6+\sum_{i=1}^{4}3^{i}=126$ event sequences. However, many of them are redundant. For instance, the sequences A;B;C;Submit and B;A;C;Submit actually address the same part of the code, hence one of them is unnecessary for testing purposes. To remedy this issue, JSDep implemented a partial-order reduction in Artemis which prunes redundant event sequences by leveraging DOM event dependencies.

To cover all code of the running example, each event handler of all three checkboxes has to be executed at least two times (examining checked/unchecked states). Therefore, a sequence of length seven (e.g., A;B;C;Submit;A;B;C) is sufficient to fully cover all the code. However, if one sets the depth bound of the test sequence to be seven for the full code coverage, the default search algorithm in Artemis and JSDep may explore at least $\sum_{i=1}^{7}3^{i}=3279$ event sequences. Notice that both Artemis and JSDep may re-execute previously executed test sequences in order to explore further the event space, which is time-consuming. Partially because of this, within a time limit these tools often generate and execute only short event sequences. Similar to classic program analysis, it is not hard to envision that short event sequences would hamper the coverage of code. Indeed in our running example, covering the function Submit requires test sequences with length at least 4. Unfortunately, existing approaches suffer from the “test sequence explosion” problem when increasing the depth bound of testing sequences. In this work, we propose a model-based, automated testing approach for JavaScript Web applications, aiming to generate long event sequences to improve the code coverage, but do so in a clever way to mitigate the issue of exponential blowup.

II-D Overview of Our Approach

Figure 3 presents an overview of our approach implemented in LJS, which consists of four components: static analysis, execution engine, model construction and testcase generation. Given the HTML/JavaScript source file(s) of a (client-side JavaScript Web) application, a length bound of event sequences (Max. Length), and a bound of the number of event sequences to be generated (#Testcases) as inputs, LJS outputs a (line) coverage report. Internally, LJS goes through the following steps: (1) compute the DOM event dependencies via static analysis. (2) construct an FSM model of the application with a variant of depth-first search by leveraging an execution engine and the DOM event dependencies. The FSM model is used to generate long event sequences. (3) all the generated event sequences are iteratively executed on the execution engine and output the coverage report. We now elaborate these steps in more detail.

Static Analysis. Given the HTML/JavaScript source file(s) of an application, this component computes the DOM event dependencies. In our implementation, we leverage JSdep [22] to compute DOM dependency. It first constructs a control flow graph (CFG) of the JavaScript code, and then traverses the CFG and encodes the control and data flows (i.e. dependency relations) in Datalog. The DOM event dependencies are finally computed via a Datalog inference engine. More details can be found in [22]. The analysis can handle dynamic registration, triggering, and removal of event handlers, but not other dynamic nature like dynamic code injection and obfuscation.

Execution Engine. Execution engine is used for FSM model construction and event sequence execution. It loads and parses the source file(s), and then executes the top-level JavaScript code. In particular, for FSM model construction, it interacts with the model constructor by iteratively receiving input events and outputting an (abstract) successor state and a set of available events at the successor after executing the input event. For event sequence execution, it receives a set of event sequences, executes them one by one and finally outputs a coverage report. For these purpose, we implement an execution engine based on Artemis [2] with its features such as event-driven execution model, interaction with DOM of web pages, dynamic event handlers detection.

Model Construction. The model constructor interacts with the execution engine by iteratively making queries to generate an FSM model up to the given length bound (i.e., Max. Length). The FSM model is intended to represent behaviors of the application. A state of the FSM model denotes an (abstract) state of the application, and a transition $(s,e,s^{\prime})$ denotes that after executing the event handler $e$ at the state $s$, the application enters the state $s^{\prime}$. The model constructor starts with an FSM containing only one initial state, explores new state $s^{\prime}$ by selecting one event $e$ available at the current state $s$, adds $(s,e,s^{\prime})$ into the FSM model, and continues exploring the state $s^{\prime}$. LJS allows to restart the exploration from the initial state and adds new states and transitions into the FSM model, in order to make the FSM model more complete.

Testcase Generation. The testcase generator traverses the FSM model to generate event sequences. LJS supports two event sequence generation algorithms: (1) partial-order reduction (POR) based event sequence generation, and (2) random event sequence generation. The first algorithm traverses the FSM model from the initial state and covers all the paths up to a (usually small) bound, while redundant event sequences are pruned based on the POR from JSDep [22]. It usually generates many short event sequences and is regarded as the baseline algorithm. The second algorithm repeatedly and randomly traverses the FSM model from the initial state to generate a small number of longer (up to a usually large bound) event sequences, and, as such, does not cover all possible paths.

To give a first impression of the performance of LJS, we ran Artemis, JSDep and LJS on the running example introduced in Section II-B. LJS reached 100% (line) coverage in 0.2s using one event sequence with length $7$, Artemis executed 3209 event sequences in 10min and reached 86%, and JSDep executed 64 event sequences in 0.7s and reached 100% coverage. We also ran JSDep and LJS on a variant of the running example which contains 10 checkboxes and the condition ${\tt count}>=3$ is replaced by ${\tt count}>=6$. LJS reached 100% coverage in 0.4s using one event sequence with length $(10\times 2+1)=21$, while JSDep executed 462 event sequences in 27.4s and reached 100% coverage. This suggests that a small number of longer event sequences could outperform a large number of shorter event sequence for applications with intensive DOM event dependency.

III Methodology

In this section, we present details of our model construction and testcase generation procedures.

III-A Model Construction

Algorithm 1 presents the model construction procedure, which takes an application $P$ and a maximum bound $d$ as inputs, and outputs an FSM model $M=(S,I,\delta,s_{0})$. It first calls the function ${\tt GetInitPage}$ which loads and parses $P$, then executes the top-level JavaScript code, finally returns the initial Web page $cur$ and the set of available events at this page which are dynamically detected. The initial state of the FSM model $s_{0}$ is obtained by calling the function ${\tt GetState}(cur)$. Intuitively, ${\tt GetState}$ computes a state from the source code of the Web page $cur$ (see below). After the initialization, Algorithm 1 iteratively selects and executes events to explore the state space up to $d$ rounds. During each iteration, it selects one available event $e$ at the current Web page $cur$ by using the ${\tt GetEvent}$ function (Line 6), based on some event selection strategy (see below). Then, it calls ${\tt GetNextPage}$ to get the next Web page $suc$ by executing $e$. The state $s^{\prime}$ of the new Web page $suc$ is also obtained by calling ${\tt GetState}$. Finally, a transition $(s,e,s^{\prime})$ is added into the FSM $M$. Variables $cur$ and $i$ are then updated accordingly. As mentioned before, LJS allows to restart the exploration of the state space from the initial state, so this process may be repeated many times.

Overall, our model construction explores state space in a depth-first fashion with a large length bound (Max. Length in Figure 3), and can avoid re-executing previously executed event sequences (like Artemis [2] and JSDep [22]) or tracking state changes (like Crawljax [3]), but not all the paths are explored for efficiency consideration.

III-A1 State Abstraction

As mentioned, states of FSM are used to represent the states of the application and are computed from Web pages via the function ${\tt GetState}$. State abstraction is crucial to describe application states. On the one hand, states of FSM should contain enough data to distinguish different application states, as unexplored application states maybe be wrongly skipped if their abstracted states were explored before, which may happen if a coarse-grained state abstraction is adopted. On the other hand, over fine-grained state abstraction may generate states which are indistinguishable wrt some coverage criteria, resulting in an explosive or even infinite state-space [28]. Therefore, the implementation of ${\tt GetState}$ requires a balance between precision and scalability.

In Artemis, the state abstraction is implemented by computing the hash value of the Web page, including Web page layout and dynamically updated DOM tree, but excluding the concrete values of CSS properties and application variables, and server-side states (unless an initial server state is provided by the user). It was demonstrated that this state abstraction is effective and efficient [2].

However, our experiments find it is still too fine-grained. Artemis typically assigns random values to attributes of DOM nodes which are taken into account in the state abstraction. In our running example, Artemis assigns a random Boolean value to the implicit attribute Value of each checkbox when testing the running example, though the value of this attribute does not affect the code coverage. (Here “implicit” means that the attribute is not explicitly given in source code, but the object has such an attribute.) The FSM model constructed using the state abstraction from Artemis is depicted in Figure 4(top-part). To prevent a prohibitively large (sometimes even infinite) state-space and improve efficiency, we use a state abstraction based on the state abstraction of Artemis but discarding the random values of the implicit attributes in the DOM tree. This allows to focus on the state changes made by executing events rather than random value assignments. The FSM model constructed by our state abstraction is depicted in Figure 4(bottom-part), which is much smaller in size.

Experimental results have confirmed that our state abstraction approach significantly reduces the size of FSM models with a comparable line coverage (cf. Section IV-C).

III-A2 Event Selection Strategies

It is common that several events are available at a Web page. Evidently the selection of events will affect the quality of the FSM model. In LJS, we implement two event selection strategies: (1) random event selection as the baseline algorithm and (2) weighted event selection. The former randomly chooses one of the available events, namely, ${\tt GetEvent}(cur)$ returns a random available event from the Web page $cur$. The latter captures the impacts of the previously executed events and DOM event dependencies. Technically we focus on:

• •

Frequency of Event Execution. In principle, all the events should be given opportunities to execute. As a result, an event which has been executed would have a lower priority to be selected in the subsequent exploration. We mention that this natural idea was already used to construct an FSM model of Android app for automated testing (e.g. [29]).

• •

DOM event dependency. Some corner-case code may be explored only by some specific event sequences due to their dependency. To expose the corner code using long event sequences, the events that depend upon the previous selected events deserve a higher chance to be selected.

In the weighted event selection strategy, each event $e$ is associated with a weight, which is adjustable dynamically at runtime. The weight of $e$ is defined as follows:

${\tt weight}(e)=\frac{\alpha_{e}\times x+\beta_{e}\times(1-x)}{N_{e}+1}$,

where $\alpha_{e}$ and $\beta_{e}$ are weight parameters, $x$ is a Boolean flag determined by DOM event dependency ($x=1$ if $e$ depends upon the previous selected event, [math] otherwise), $N_{e}$ is number of times that $e$ has been executed.

With the weighted event selection strategy, ${\tt GetEvent}(cur)$ randomly returns one of events which have the highest weight among all available events at the Web page $cur$. In our experiments, $\alpha_{e}$ and $\beta_{e}$ are set to be $0.7$ and $0.3$ respectively, which are the best configuration after tuning.

Recall the example in Figure 1. At the initial state, all the available onclick events of A,B,C have the same weight $0.3$ (note that the onclick event of Submit is not available therein), i.e., they have the same chance to be selected. Suppose the onclick event of A is selected, the weights of the onclick events of A,B,C are updated to $\frac{0.7}{2},\frac{0.7}{1},\frac{0.7}{1}$ respectively. LJS then randomly chooses one of the onclick events of B,C. Suppose the onclick event of B is selected on this occasion. The weights of all the available onclick events of A,B,C become $\frac{0.7}{2},\frac{0.7}{2},\frac{0.7}{1}$, which implies that the onclick event of C will be selected at the next step. After that, the weights of the onclick events A,B,C,Submit are updated to $\frac{0.7}{2},\frac{0.7}{2},\frac{0.7}{2},\frac{0.7}{1}$ (note that the onclick event of Submit now becomes available).

III-B Testcase Generation

In this work, as mentioned in the introduction, we focus on the event sequence generation while the input data is chosen randomly. We first present the baseline algorithm for generating event sequences with partial-order reduction (POR) which is inspired by [22, 30], and then discuss how to generate long test sequences.

III-B1 Baseline Event Sequence Generation with POR

Given the FSM $M=(S,I,\delta,s_{0})$, a bound $d$, and the DOM event dependency relation $\rightarrow$. Algorithm 2 (excluding Lines 17 and 20) generates all possible event sequences with length up to $d$ stored in $T$.

The procedure Explore traverses the FSM $M$ in a depth-first manner, where $ss$ is a working stack storing the event sequence with the initial state $s_{0}$ as the bottom element, $E$ denotes the set of available events at state $s$, $s.{\tt SelectedEvent}$ denotes the selected event at $s$, and $s.{\tt done}$ denotes the set of all previously selected events at $s$. The procedure Explore first checks whether the bound $d$ is reached (Line 8), If $ss.{\tt Length}()>d$, then the while-loop is skipped. After that, if $s.{\tt SelectedEvent}={\tt null}$ (indicating that the sequence in $ss$ has reached the maximum length $d$), the event sequence stored in the stack $ss$ is added to the set $T$. Otherwise, the while-loop will explore a previously unexplored event and invoke the procedure Explore recursively. The while-loop terminates when all the available events at state $s$ have been explored. Note that in this case, $s.{\tt SelectedEvent}\neq{\tt null}$, hence the sequence in $ss$ will not be added to $T$.

With Lines 17 and 20 Algorithm 2 implements the partial-order reduction based on the notion of sleep-set [31]. In principle, it first classifies the event sequences into equivalence classes, and then explores one representative from each equivalence class. In detail, each event $e$ explored at a state $s$ is put into its sleep set $s.{\tt sleep}$ (Line 18). When another event $e^{\prime}$ is explored at $s$, the sleep set of $s$ is copied into the sleep set of the next state $s^{\prime}$ (Line 15), if the DOM events $e$ and $e^{\prime}$ are independent of each other. Later, each event at $s$ will be skipped if it is in the sleep set of $s$ (Line 11), because executing this event is guaranteed to reach a previously explored state.

III-B2 Long Event Sequence Generation

Algorithm 3 shows the pseudocode of our random event sequence generation. Given the FSM model $M=(S,I,\delta,s_{0})$, a maximum length bound $d$ and a maximum bound $m$ of the number of event sequences, Algorithm 3 randomly generates $m$ number of event sequences with length $d$. Each iteration of the outer while-loop computes one event sequence with length $d$ until the number of event sequences reaches $m$. In the inner while-loop, it starts from the initial state $s_{0}$ and an empty sequence $\rho=\epsilon$. At each state $s$, the inner while-loop iteratively and randomly selects a pair $(e,s^{\prime})$ of event and state denoting that executing the event $e$ at the state $s$ moves to the state $s^{\prime}$, then appends $e$ to the end of previously computed sequence $\rho$. Algorithm 3 repeats this procedure until the length of $\rho$ reaches the maximum bound $d$. At this moment, one event sequence is generated and stored into the set $T$. The outer while-loop enters its next iteration.

Algorithm 3 may not generate all the possible event sequences, and may generate redundant event sequences, but less often, due to large maximum bound. We remark that the POR technique cannot be integrated into Algorithm 3.

IV Experiments

We have implemented our method as a software tool LJS. It exploits JSDep [22] for computing DOM event dependency and a modified automated testing framework Artemis [2] as the execution engine. For comparison purpose, we implemented LJS in such a way that individual techniques are modularized and can be enabled on demand. Thus, we were able to compare the performance of various approaches with different configurations, in particular, (1) our state abstraction vs the state abstraction from [2], (2) random event selection vs weighted event selection, (3) baseline event sequence generation with POR (i.e., Algorithm 2) vs long event sequence generation (i.e., Algorithm 3). To demonstrate the efficiency and effectiveness of LJS, we compared LJS with JSDep [22] on same benchmarks. (We note that in [22] JSDep is shown to be superior to Artemis [2], so a direct comparison between LJS and Artemis [2] is excluded.)

The experiments are designed to answer the following research questions:

1. RQ1.

How efficient and effective is LJS compared with JSDep [22]? 2. RQ2.

How effective is our coarse-grained state abstraction compared with the state abstraction from [2]? 3. RQ3.

How effective is the weighted event selection strategy compared with the random event selection strategy? 4. RQ4.

How effective is the long event sequence generation compared with the baseline algorithm?

IV-A Evaluation Setup

To make comparison on a fair basis, we evaluated LJS on publicly available benchmarks111https://github.com/ChunghaSung/JSdep. of JSDep [22], which consist of 21 client-side JavaScript Web applications with 18,559 lines of code in total. Columns 1-2 of Table I show the name of the application and the number of lines of code. We ran all experiments on a server with a 64-bit Ubuntu 12.04 OS, Intel Xeon(R) E5-2603v4 CPU (1.70 GHz, 6 Cores), and 32GB RAM. To answer the research questions RQ1-RQ4, we conducted four case studies. The time used to compute the DOM event dependency is usually marginal and can be safely ignored, so is not counted in line with [22]. For statistics, we ran LJS on each application 5 times and simply took the average as the result. The coverage measure are the aggregation of that from model construction and testcase execution respectively which are separated in the last experiment.

IV-B RQ1:* LJS vs. JSDep*

In this study, we answer RQ1 by performing two experiments assessing effectiveness and efficiency. For effectiveness, we compare the (line) coverage of LJS with JSDep in 600 seconds. For efficiency, we compare the time used by LJS and JSDep to achieve the same coverage. The experiments of LJS were performed with the following configurations: Max. Length=99, random event selection strategy, our new state abstraction, running Algorithm 1 two times, and long event sequence generation. Experiments of JSDep were performed with the setting shown in [22].

Table I shows the results of LJS and JSDep in 600s, where Columns 3-6 (resp. Columns 7-9) show the average, standard deviation (sd) of coverage obtained in 5 runs, number and length of event sequences after running LJS (resp. JSDep).

Overall, we can observe an increase in the average coverage from 81% achieved by JSDep to 86.4% achieved by LJS. (We remark that the average coverage was increased from 67% achieved by Artemis to 80% achieved by JSDep in 600s [22]. In our experiment, JSDep performed slightly better.) Perhaps more importantly, on large applications such as frog, sokoban, 3dmodel, cubuild and ball_pool, the coverage of LJS is 11-18% higher than that of JSDep. We can also observe that a small amount of long event sequences could outperform a large amount of short event sequences. However, it should be emphasized that long event sequences, but of low quality, may not improve the coverage. This has been demonstrated by the results of case4, speedyeater and fullhouse.

Figure 5 shows the coverage that LJS and JSDep achieved by running on the top 6 largest applications in time ranging from 60s to 600s with step size 60s. (Note that the application lady is excluded in this experiments because its model construction takes more than 600s; cf. Table I.) The X-axis is the execution time budget whereas Y-axis is the achieved coverage. Overall, as the execution time budget increases, the coverage of LJS is higher than that of JSDep. Moreover, the rate of LJS to achieve a higher coverage is, in most cases, slightly higher than that of JSDep.

IV-C RQ2:* Comparison of State Abstraction Techniques*

In this study, we answer RQ2 by performing one experiment and comparing the obtained FSM model and coverage results using our coarse-grained state abstraction and the state abstraction from [2] respectively. In this experiment, LJS constructs the FSM model by running Algorithm 1 two times using Max. Length=99 and random event selection, and generates two event sequences from the FSM model. Taking into account the consumption of time, the experiment only generates two event sequences.

Table II shows the results, where Columns 2-5 (resp. Columns 6-9) show the coverage, numbers of states and transitions of the FSM model (note that we take the average of these numbers), and execution time after running LJS with the state abstraction from [2] (resp. our new state abstraction).

Overall, the numbers of states and transitions using our state abstraction are much smaller, with a dramatic decrease in some large applications such as sokoban, 3dmodel, pearlski, speedyeater and harehound. Meanwhile, the performance of the two state abstractions (in terms of average coverage and execution time) is comparable.

IV-D RQ3:* Comparison of Event Selection Strategies*

In this study, we answer RQ3 by performing one experiment and comparing the obtained FSM model and coverage results using weighted event selection strategy and random event selection strategy respectively. In this experiment, LJS constructs the FSM model by running Algorithm 1 two times using Max. Length=99 and our coarse-grained state abstraction, and generates two event sequences from the FSM model.

Table III shows the results, where Columns 2-5 (resp. Columns 6-9) show the coverage, numbers of states and transitions of the FSM model, and execution time after running LJS with random event selection (resp. weighted event selection).

Overall, the numbers of states and transitions of weighted event selection are respectively less than that of random event selection. Meanwhile, the average coverage and execution time of the two selection strategies are comparable. In particular, in terms of coverage for applications such as cubuild the weighted event selection strategy performs better, whereas for applications such as fullhouse, the random event selection strategy performs better. We will discuss these findings later.

IV-E RQ4:* Comparison of Event Sequence Generations*

In this study, we answer RQ4 by performing two experiments and comparing quality of event sequences generated by the long event sequence generation (Algorithm 3) and baseline event sequence generation with POR (Algorithm 2). In the first experiment, LJS constructs the FSM model by running Algorithm 1 two times using our coarse-grained state abstraction and the random event selection strategy, and generates two event sequences from the FSM model, while the maximum bounds (i.e., Max. Length) are 9, 39, 69, 99, 129 and 159. In the second experiment, we first generate an FSM model using the same configuration with a fixed maximum bound 99. For each application, from the same FSM model, LJS with Algorithm 3 enabled iteratively generates and executes event sequences with maximum bound 99 within 600s time bound. Meanwhile, LJS with Algorithm 2 enabled generates and executes all event sequences with redundant sequences pruned up to some maximum bound so that the execution time exceeds 600s, moreover, the execution terminates when the execution time reaches 1200s.

Figure 6 shows the results of the first experiment, where the X-axis is the maximum bound, and Y-axis is achieved coverage with that bound, and the black bold line shows the trend of average coverage.

Overall, the average coverage increases quickly when the maximum bound increases from 9 to 39, but only slightly when it increases from 39 to 159.

Table IV shows the results of the second experiment, where Columns 2-5 (resp. Columns 6-9) show the coverage (of testcase execution only), maximum sequence length, execution time and number of event sequences after running LJS with Algorithm 2 (resp. Algorithm 3) enabled. Overall, the average coverage of Algorithm 3 (i.e., long event sequence generation) is 6.5% higher than that of Algorithm 2 (i.e., baseline event sequence generation with POR) with less execution time. In particular, the coverage improvement of Algorithm 3 is more prominent for applications case2-case4, chinabox, 3dmodel and cubuild. These results also confirm that executing fewer long event sequences may achieve higher coverage than executing more short event sequences.

IV-F Discussions

The coverage improvement, as shown in the experiment, is not significant using the weighted event selection. There are two possible reasons. First, weighted event selection relies upon DOM event dependencies which are computed by JSDep. JSDep uses context, path, flow and object-insensitive static analysis, hence the dependencies may be not sufficiently fine-grained. Using more accurate DOM event dependencies may improve the effectiveness of the weighted event selection. Second, the selected benchmarks might not be representative of real-world JavaScript programs that are DOM event dependency intensive. LJS currently supports off-line testing (i.e. when source code is available) when DOM event dependency is enabled. Our approach is also applicable in on-line testing if the DOM event dependency is computed dynamically, as it does not need to record and replay.

We note some limitations of the experiments. The experiments are based on a benchmark that includes only 4 large-scale Web applications (with more than 1k LOC, maximum 5k LOC). More experiments are needed to assess randomized algorithms, for instance, the statistical significance of the coverage improvement [32].

V Related Work

We discuss the related work in the areas of model-based testing and automated JavaScript Web application testing.

V-A Model-based Automated Testing

Model-based testing (MBT) has been widely used in software testing (cf. [33, 34, 35] for surveys). Mainstream MBT techniques differ mainly in three aspects: models of the software under test, model construction, and testcase generation. Several models, such as state-based (e.g., pre-/post-condition) and transition-based (e.g., UML and I/O automata), have been proposed. The FSM model used in this work is one of the transition-based models. Model construction is one of the most important tasks in MBT. It is usually time-consuming and error-prone to manually construct models for GUI-based applications [36, 37]. Therefore, most works use static/dynamic analysis to construct models, for example, [38, 39, 40, 29, 27, 26, 41, 42] for mobile/GUI applications. However, it is rather difficult to statically construct models for JavaScript Web applications due to their dynamic characteristics [3]. Regarding works on model construction for JavaScript Web applications [43, 44, 45, 46, 3, 47]; [43, 46, 44] have to construct a model manually. [43] extracts the model via static analysis, but lacks of considering dynamic nature of JavaScript; [45, 3] construct FSM models via dynamic analysis to crawl Web applications. The main difference between our work and theirs [45, 3] is the way in which the model is constructed. Our model construction pursues larger depth without backtracking, but does not cover all possible event sequences, whereas [45, 3] cover all the possible event sequences up to a length bound with backtracking. [47] also reported to construct FSM models, but did not give detail of their algorithm, nor included JavaScript coverage. Existing testcase generation algorithms mainly focus on the systematic generation of event sequences with a rather limited length bound due to “test sequence explosion” problem. Our approach generates long event sequences, but strategically avoid covering all possible event sequences (up to a length bound) to mitigate the exponential blowup problem. The tool Stoat [29] considered model-based testing for Android apps, but with different testcase generation strategy as ours.

V-B JavaScript Web Application Automated Testing

Web application testing has been widely studied in the past decade, differing mainly in targeted Web programming languages (e.g., PHP [48, 49, 50] and JavaScript [2, 3, 5]), and testing techniques (e.g., model-based testing [33, 34, 35], mutation testing [15, 7, 46], search-based testing [48, 51, 52, 53] and symbolic/concolic testing [1, 6, 12]; cf. [28, 21] for surveys). We mainly compare with works on automated testing of JavaScript Web Application.

Test sequences of JavaScript Web Applications consist of event sequences and input data for each event. Existing works create event sequences via exploring the state space by randomly selecting events with heuristic search strategies [1, 2, 13, 3, 10]. For instance, Kudzu [1] and Crawljax [3] randomly select available events. Moreover, Crawljax relies on a heuristic approach for detecting event handlers, hence may not be able to detect all of them. Artemis [2] uses the heuristic strategy based on the observed read and write operations by each event handler in an attempt to exclude sequences of non-interacting event handler executions. EventBreak [13] uses the heuristic strategy based on performance costs in terms of the number of conditions in event handlers in an attempt to analyze responsiveness of the application. These approaches usually cover all the sequences up to a given, usually small length bound. In order to explore long event sequences in limited time, delta-debugging based method [54] and partial-order reduction [22] were proposed for pruning redundant event sequences. Our approach does not cover all possible event sequences, hence can create long event sequences within the time budget. Experimental results show that our approach can achieve a high line coverage than Artemis even with partial-order reductions [22].

Another research line in automated testing of JavaScript Web application is to generate high quality input data of events using symbolic/concolic testing, e.g., [1, 6, 12, 16, 17, 19]. These approaches are able to achieve high coverage, but heavily rely on the underlying constraint solver. They generally do not scale well for large realistic programs, because the number of feasible execution paths of a program often increases exponentially in the length of the path. Our work focuses on the generation of long event sequences, but choose the input data randomly, which is orthogonal, and could be complementary, to the more advanced input data generation methods. A transfer technique has been proposed based on the automation engine framework Selenium to transfer tests from one JavaScript Web application to another [55]. This is orthogonal to this work.

VI Conclusion and Future Work

We have proposed a model-based automated testing approach for JavaScript Web applications. Our approach distinguishes from others in making use of long event sequences in both FSM model construction and testcase generation from the FSM model. We have implemented our approach in a tool LJS and evaluated it on a collection of benchmarks. The experimental results showed that our new approach is more efficient and effective than Artemis and JSdep. Furthermore, we empirically found that longer test sequences can achieve a high line coverage.

For future work, we plan to experiment on more larger benchmarks and to have a thorough statistical analysis. In particular, the current evaluation is based on coverage, but it would also be interesting to evaluate the fault detection. Furthermore, we note that there are several ways to obtain long sequences based on FSM, for instance, by following the dependency chains or giving priority to the novel states. These strategies deserve further exploration.