EVMFuzz: Differential Fuzz Testing of Ethereum Virtual Machine

TL;DR

EVMFuzz is a differential fuzz testing tool that detects vulnerabilities in Ethereum Virtual Machines by generating and mutating smart contracts to find inconsistencies across different EVM implementations.

Contribution

The paper introduces EVMFuzz, a novel differential fuzz testing approach for EVMs, including new metrics, seed contract generation, and cross-referencing techniques to identify security bugs.

Findings

Detected 1,596 inconsistent contracts among EVMs

Discovered 5 previously unknown security bugs

Validated bugs are listed in CVE database

Abstract

Ethereum Virtual Machine (EVM) is the run-time environment for smart contracts and its vulnerabilities may lead to serious problems to the Ethereum ecology. With lots of techniques being developed for the validation of smart contracts, the security problems of EVM have not been well-studied. In this paper, we propose EVMFuzz, aiming to detect vulnerabilities of EVMs with differential fuzz testing. The core idea of EVMFuzz is to continuously generate seed contracts for different EVMs' execution, so as to find as many inconsistencies among execution results as possible, eventually discover vulnerabilities with output cross-referencing. First, we present the evaluation metric for the internal inconsistency indicator, such as the opcode sequence executed and gas used. Then, we construct seed contracts via a set of predefined mutators and employ dynamic priority scheduling algorithm to guide seed contracts selection and maximize the inconsistency. Finally, we leverage different EVMs as crossreferencing oracles to avoid manual checking of the execution output. For evaluation, we conducted large-scale mutation on 36,295 real-world smart contracts and generated 253,153 smart contracts. Among them, 66.2% showed differential performance, including 1,596 variant contracts triggered inconsistent output among EVMs. Accompanied by manual root cause analysis, we found 5 previously unknown security bugs in four widely used EVMs, and all had been included in Common Vulnerabilities and Exposures (CVE) database.

Figures

Tables

Table 1. Table 1. Typical Mutators

Object	Description of Mutator	Example (contract source code before mutation $\to$ contract source code after mutation)
local variable	Select a method and modify one or more local variable attribute(s)	function multiPath (uint a, uint b) public returns (uint) { \markoverwith \ULonuint x; } $\to$ function multiPath (uint a, uint b) public returns (uint) { \markoverwith \ULonbytes x; }
function property	Select a method and insert, modify or delete its function attribute(s)	function multiPath (uint a, uint b) \markoverwith \ULonpublic returns (uint) {…} $\to$ function multiPath (uint a, uint b) \markoverwith \ULonconstant returns (uint) {…}
arithmetic operator	Select a method and modify one or more inner arithmetic operator(s)	function multiPath (uint a, uint b) public returns (uint) { uint c = \markoverwith \ULona + b; } $\to$ function multiPath (uint a, uint b) public returns (uint) { uint c = \markoverwith \ULona - b; }
conditional operator	Select a method and modify one or more inner conditional operator(s)	… if (\markoverwith \ULona > b) { … } … $\to$ … if (\markoverwith \ULona <= b) { … } …
loop operator	Select a method and modify one or more inner loop execution statement(s)	… for (uint i = 0; \markoverwith \ULoni < 100; ++i) { … } $\to$ … for (uint i = 0; \markoverwith \ULoni < 199; ++i) { … }
assert statement	Select a method and insert or delete one or more assert statement(s)	function sub (uint a, uint b) public returns (uint) { … } $\to$ function sub (uint a, uint b) public returns (uint) { … \markoverwith \ULonassert (a > b); … }
return statement	Select a method and delete the return statement	function get (uint a) public \markoverwith \ULonreturns (uint) {… \markoverwith \ULonreturn a; …} $\to$ function get (uint a) public { … }
control structure	Select a method and insert continue or break statement in loop structure	… for (uint i = 0; i < 10; ++i) { a += b; } $\to$ for (uint i = 0; i < 10; ++i) { a += b; \markoverwith \ULonbreak; }

Table 2. Table 2. gasUsed inconsistency

	js-evm	Py-EVM	aleth	geth	total	percent
js-evm	0	18115	31166	17486	66767	66.59%
Py-EVM	18115	0	31176	1358	50649	50.51%
aleth	31166	31176	0	31163	93505	93.25%
geth	17486	1358	31163	0	50007	49.87%

Table 3. Table 3. opcode inconsistency

	js-evm	Py-EVM	aleth	geth
js-evm	0	1275	52	52
Py-EVM	1275	0	1240	1240
aleth	52	1240	0	0
geth	52	1240	0	0

Table 4. Table 4. output inconsistency refinement

	js-evm	Py-EVM	aleth	geth
value of Ind.1	131	0	38	0
value of Ind.2	9	14	0	0
total inconsistency	140	14	38	0

Table 5. Table 5. Final weights of each mutator

Mutator	Object	Method	Weight
#2	conditional operator	Modify all conditional operators	0.2703
#3	arithmetic operator	Modify all arithmetic operators	0.1691
#5	loop operator	Increase the iteration of loop by 99	0.1180
#8	control structure	Insert continue or break statement	0.1043
#1	local variable	Modify all variable attributes	0.1022
#6	assert statement	Delete or insert an assert statement	0.0963
#4	function property	Modify all function attributes	0.0765
#7	return statement	Delete the return statement	0.0631

Table 6. Table 6. Description of 5 high-risk vulnerabilities detected by EVMFuzz

CVE-ID	platform	version	language	description	created date
CVE-2018-18920	Py-EVM	v0.2.0-alpha.33	python	Py-EVM v0.2.0-alpha.33 allows attackers to make a vm.execute_bytecode call that triggers illegal values shown in stack.	20181103
CVE-2018-19183	js-evm	v2.4.0	JavaScript	ethereumjs-vm 2.4.0 allows attackers to cause a denial of service (vm.runCode failure and REVERT) via a "code: Buffer.from(my_code, ’hex’)" attribute.	20181111
CVE-2018-19184	geth	v1.8.17	golang	cmd/evm/runner.go in Go Ethereum (aka geth) 1.8.17 allows attackers to cause a denial of service (SEGV) via crafted bytecode.	20181111
CVE-2018-19330	aleth	v1.5.0-alpha.6	cpp	** RESERVED ** Details would be public after the vulnerability has been repaired to avoid potential attack.	20181117
CVE-2019-7710	aleth	v1.5.0-alpha.7	cpp	** RESERVED ** Details would be public after the vulnerability has been repaired to avoid potential attack.	20190210

Equations

$$gasDiff(i,j)=norm(abs(gasUsed(i,C),gasUsed(j,C)))$$

$$opDiff(i,j)=norm(abs(opSeqLen(i,C),opSeqLen(j,C)))$$

$$diff=gasDiff(i,j)+opDiff(i,j)$$

$$outVul(i,j)=output(i,C)\oplus output(j,C)$$

Full text

In this paper, we propose EVMFuzz, aiming to detect vulnerabilities of EVMs with differential fuzz testing. The core idea of EVMFuzz is to continuously generate seed contracts for different EVMs’ execution, so as to find as many inconsistencies among execution results as possible, eventually discover vulnerabilities with output cross-referencing. First, we present the evaluation metric for the internal inconsistency indicator, such as the opcode sequence executed and gas used. Then, we construct seed contracts via a set of predefined mutators and employ dynamic priority scheduling algorithm to guide seed contracts selection and maximize the inconsistency. Finally, we leverage different EVMs as cross-referencing oracles to avoid manual checking of the execution output. For evaluation, we conducted large-scale mutation on 36,295 real-world smart contracts and generated 253,153 smart contracts. Among them, 66.2% showed differential performance, including 1,596 variant contracts triggered inconsistent output among EVMs. Accompanied by manual root cause analysis, we found 5 previously unknown security bugs in four widely used EVMs, and all had been included in Common Vulnerabilities and Exposures (CVE) database.

Differential testing, fuzzing, domain-specific mutation, EVM

Introduction

Ethereum can be viewed as a transaction-based state machine (Wood, 2014), whose foundation is transaction execution. There are about 500,000 transactions (Etherscan, 2018b) running on Ethereum every day, most of which involve the execution of smart contracts. Over the past few years, the safety and security problems of the blockchain transaction caused by smart contracts have emerged endlessly. In June 2016, Blockchain industry’s largest crowdfunding project — $TheDAO$, was attacked due to a serious flaw in $splitDAO$ function, resulting in more than three million Ether loss (WIRED, 2018). Then, many efforts are devoted to safeguarding the transaction security by ensuring rigorous code logic of smart contracts. For instance, Oyente (Luu et al., 2016) and MAIAN (Nikolic et al., 2018) use symbolic execution techniques to find potential security vulnerabilities in Solidity smart contracts. Zeus (Kalra et al., 2018) applies abstract interpretation to analyze smart contracts.

However, as the authentic platform and standard for executing smart contracts, if there are some vulnerabilities in EVM’s internal implementation, it will definitely lead to serious consequences such as Ethereum operation errors or transaction failures. We selected 10 contracts that passed Oyente (Luu et al., 2016) tool examination and executed them on EVMs implemented by JavaScript, Python, C++ and Go language, finding that 7 of them had different gas consumption on different platforms at the end of execution, and one function even had the result of execution failure on Py-EVM (Ethereum, 2018c), which is the Python implementation of the EVM. This shows that only contract validation cannot ensure the correctness of Ethereum transactions execution, and it is of great urgency to secure EVM. To perform efficient EVM testing, we need to solve the following challenges:

• •

**How to define general scenarios and evaluation metrics? **  

At present, EVM has at least 10 widely used implementations of different programming language (Ethereum, 2018), all of which are based on the standards of Ethereum Yellow Paper (Wood, 2014). For example, the amount of EVM code in Geth (Ethereum, 2018a) platform is about 5,475 lines of Go, accounting for 27.6% of the core infrastructure implementation of Geth; the data in Parity (Ethereum, 2018b) is about 18,912 lines of Rust, accounting for 31.7% of the core infrastructure implementation. The implementation of EVM is complex, involving a large number of control structures and storage structures. Because of the EVM version diversity and code complexity, it is necessary to define meaningful and general evaluation metric for the testing of different EVMs.

• •

**How to generate test cases that trigger EVM bugs? **  

The Ethereum Yellow Paper provides the basis for all implementations of EVM, in which the functions and attributes are defined by formulas and rules. But there is neither authoritative test suites and benchmarks nor the common vulnerabilities and defects of EVM platforms sorted out by authority, which makes EVM testing loses the basis and target of detection. Moreover, there is no mature testing tool for EVM which makes it difficult to employ EVM testing on a large scale in the short term. Due to the lack of official benchmarks and widely used EVM testing frameworks, we urgently need to find out a solution which is able to perform efficient and accurate EVM testing, automatically generating EVM inputs fast and effective.

To address these challenges, we implement EVMFuzz, which aims to automatically generate adversarial test inputs for EVMs.

First, we define a general evaluation metric for the differential fuzzing of EVMs. As most EVMs are implemented as a transaction-based state machine, and the change of state depends on the opcode sequence to be executed, the input parameters and gas limit, hence, we use the opcode sequence executed and gas used as two important indicators to evaluate EVMs’ performance on each test contract. EVMFuzz integrates different EVMs and creates a unified running environment for them. In this way, it takes the natural advantages of multiple versions to quickly discover the output inconsistencies without manual checking. Then, our seed contract mutation and selection algorithms can continuously generate contracts that enlarge the metric difference, so that EVMFuzz can efficiently mine cases that trigger differential performance of EVMs and try to get those corner cases with inconsistent execution output.

For evaluation, we firstly conducted empirical studies on 36,295 real-world smart contracts from Etherscan (Etherscan, 2018a) and found that 24,000 contracts triggered metric inconsistencies among different EVMs, even with the same execution output. Through guided fuzzing, 1,596 variant contracts successfully triggered inconsistent execution output among different EVMs. With manual analysis, we found 5 previously unknown security bugs in different widely used EVMs, and all had been included in Common Vulnerabilities and Exposures (CVE) database (MITRE, 2018a).

Contributions  We make the following main contributions:

• •

We introduce an evaluation metric for EVM differential fuzzing, define 8 mutators for seed contract generation and design dynamic priority scheduling algorithm for seed contract selection.

• •

We implement EVMFuzz, an automated differential fuzz testing framework, to efficiently expose the differences and vulnerabilities of different EVMs.

• •

We apply EVMFuzz to test some most widely used EVM versions, many inconsistencies and security bugs are detected, and 5 vulnerabilities have been assigned with unique CVE IDs.

Paper Organization  The rest of this paper is organized as follows. Section 2 introduces some background and gives a motivating example. We provide a high-level overview of EVMFuzz in Section 3, and the details implementation are described in Section 4. The evaluation results are shown in Section 5. Section 6 outlines the limitations and proposes future directions for improvement. Finally, we survey related work in Section 7 and conclude in Section 8.

Background

2.1. The Ethereum Virtual Machine

Ethereum Virtual Machine (EVM) is the heart of the Ethereum, which is often called the operating system of the Ethereum technology and is responsible for the execution and maintenance of smart contracts. It is the bedrock on which smart contracts are built.

The formal definition of the EVM is specified in the Ethereum Yellow Paper (Wood, 2014). EVM is a simple stack-based architecture, whose word size (size of stack items) is 256-bit. According to the predefined execution environment and execution steps, such as exception halting and jump destination validity, it completes the state transition of each Ethereum block. EVM handles the execution of bytecode and the calculation of gas consumption. In general, EVM is a powerful, sandboxed virtual stack embedded within each full Ethereum node, responsible for executing contract bytecode.

In accordance with the standards of Ethereum Yellow Paper, EVMs have been successfully implemented in various programming languages including C++, Go and many others (Ethereum, 2018). There are tens of thousands of people doing transactions via the clients based on these EVM implementations everyday. Therefore, the vulnerability hidden in any EVM version might result in serious consequences.

2.2. A Motivating Example

To investigate the effectiveness of EVMFuzz, we use a simple example presented below. $forTest$ is a simple contract with a function $Testwhile$, whose data structure is a $while$ loop. When the input parameter $a$ is less than $b$, variable $x$ will continually increase. Strictly speaking, the implementation of $Testwhile$ function has a serious problem and may not appear in real life, if the input parameter satisfies the loop condition, it will result in an infinite loop. However, such contract can pass the check of most existing contract testing and verification tools, and our experiment proves that it can trigger different behaviors of multiple EVM implementations and even cause the denial of service problem.

⬇

1pragma solidity ^0.4.24;

2contract forTest {

3 function TestWhile (uint a, uint b) public {

4 uint x = 1;

5 while (a < b) {

6 x += 1;

7 }

8 }

9}

We test the contract on three different EVM versions—JavaScript implementation js-evm (EthereumJS, 2018), Python implementation Py-EVM (Ethereum, 2018c), and C++ implementation aleth (Ethereum, 2018). When the $TestWhile$ function is called with parameters $a=1$, $b=2$, js-evm will continually print trace information, together with the increase of CPU usage and terminate the operation correctly; Py-EVM will print trace information and terminate the operation correctly; aleth will not print trace information, but continue to occupy CPU resources until the system kills the whole process and crashes. Comparing the execution results, we find that aleth can neither handle the extreme situation of infinite loop properly that has no friendly user interaction or information feedback nor cut the loss in time and reduce malicious resource occupied, eventually leads to potential denial of service.

This example shows that some contracts containing corner cases can trigger the boundary condition of EVM implementations and expose unexplored defects. But these contracts involving extreme circumstances are often inconsistent with logic programming rules, which require artificial construction, in other word, contract mutation. Furthermore, for massive mutated contracts, test oracle is difficult to artificially define, and some extreme cases are not designated in EVM design specifications. Therefore, it is an efficient and effective way to apply differential fuzzing.

Approach Overview

In this section, we briefly introduce the workflow of EVMFuzz. Our goal is to apply differential fuzz testing on EVMs. The concept of differential fuzz testing is very simple, that is, to continuously provide invalid, unexpected or random data as inputs to several programs with the same functions. These programs are then monitored for catching "different act" on some inputs, if so, we may find a bug in some of the programs. In this paper, our test object are the same functional EVM platforms implemented by different programming languages, and the test input is the mutated smart contract. An overview of EVMFuzz is given in Fig. 1, which consists of two major components, i.e., seed contract generation based on static analysis and unified EVM execution based on fuzzing loop. We will also introduce the evaluation metrics for EVM differential fuzzing.

3.1. Seed Contract Generation

The input for the seed generation module is the smart contract file, and the output is a contract variant whose key property has been modified by specific mutators. First, we precisely construct the Critical locations identified Abstract Syntax Tree (CAST) of the seed contract (§4.1), for facilitating subsequent mutation and analysis. Then the seed contract will be put into the seed pool. EVMFuzz will rank the candidate contracts as a prioritized queue under the guidance of dynamic priority, and the contract in the first place will be selected as the next subject (§4.2). After choosing the contract for mutation, EVMFuzz uses 8 predefined mutators and the combined strategy to guide mutation (§4.3) and obtains the input for unified EVM execution module. The goal is to generate contracts that can increase the degree of metric difference and trigger different execution output.

3.2. Unified EVM Execution

EVM execution module provides a unified runtime environment for various EVMs (§4.4). After receiving the contract file from the seed generation module, it compiles the seed into EVM bytecode. The input parameter is generated according to the data type of the called function, thus the uniform input for each EVM is obtained. Then EVMFuzz automatically runs all EVMs, calculates the difference information according to the test metric, and compares the execution output results. Finally, according to the seed’s ability to enhance the degree of metric difference, EVMFuzz decides whether to put the seed contract into the seed pool where high-quality seeds preserved (§4.5). Besides, when the execution output is inconsistent, this module will also record the potential exception for manual root cause analysis.

3.3. Metrics Formulation

To evaluate the performance of each EVM on the test contract, we define the metric on two general indicators. As most EVMs are implemented as a transaction-based state machine, and the change of state depends on the sequence of opcode to be executed, the input parameter and gas limit, hence, we use the internal opcode sequence executed and gas used as the two indicators.

1. (1)

opcode sequence. Opcode is short for operation code, which is used to describe the part of machine code that performs some sort of operations in machine language instructions. From the perspective of computer instruction execution, each function call is completed by a series of opcode execution. The opcode sequence clearly shows a complete process of contract operation, which can be used to check the execution correctness of each step. For platform $i$, we define $opSeqLen(i,C)$ as the length of opcode sequence of $i$ when executing contract $C$. 2. (2)

gasUsed. $gasUsed$ is the total number of gas consumed by all operations in a transaction or message. The value of $gasUsed$ is vitally interrelated with the success of transaction execution, and is also directly related to the transaction fee that users ultimately need to pay. Here we use $gasUsed(i,C)$ to represent total gas consuming of platform $i$ after running contract $C$.

Based on these two indicators, we further define the evaluation metric of difference information. When given an input parameter, the normal execution of a transaction on a dedicated EVM platform is determined by a confirmed and unique execution sequence, and the total gas consumption is also calculated. Therefore, we construct an evaluation metric $diff$ to measure the difference among different EVMs execution(§4.5). The greater the metric difference, the higher probability the inconsistent execution output. Execution output is the return value after all executions, that $output(i,C)$ is defined as the returns of $C$’s execution on EVM $i$. For a function call, it is the returned data, and for a transaction, it is the balance. While the metric defined on the two internal indicators reflects the implementation and execution difference of different EVMs, the execution output can intuitively reflect whether those EVMs are running consistently or correctly.

EVMFuzz Design

In this section, we will elaborate on the key components in Fig. 1.

4.1. CAST Construction

Before EVMFuzz starts the entire procedure of fuzzing, it first carries out static analysis on initial seed contracts and generates the CAST structure for further mutation.

A CAST of a smart contract is a structured tree representation of the abstract syntactic structure of Solidity source code. Each node of the tree denotes a construct occurring in the source code. CAST can define and decompose properties in all statements of contract. Transforming a contract into CAST structure can help us complete the subsequent contract mutation operations. It can directly search, replace, delete or insert operators according to the key attributes.

Furthermore, CAST identifies critical locations of a seed contract, which are the subtree of statements related to ether transaction. It mainly involves six statement symbols — $new$, $call$, $delegatecall$, $callcode$, $send$ and $transfer$. Based on CAST, we can guide pre-defined mutators to select the structures that are identified as critical locations in order to test the core functions of EVM. A simple example is presented in Fig.2, including the source code and the corresponding CAST structure, where the shaded nodes are regarded as the critical locations, for the reason that they are all under the subtree of the $call$ statement.

⬇

1pragma solidity ^0.4.24;

2contract Demo {

3 function transfer (address from,address caddress,address[] _tos,uint v) public returns (bool) {

4 require(_tos.length > 0);

5 bytes4 id=bytes4(keccak256(

6 "transferFrom(address,address,uint256)"));

7 for(uint i=0;i<_tos.length;i++) {

8 caddress.call(id,from,_tos[i],v);

9 }

10 return true;

11 }

12}

4.2. Seed Contract Prioritization

In seed contract pool, the importance of each candidate contract is different. In general, the contract that makes the metric difference among EVMs larger should be the benchmark for the next mutation iteration. But at the same time, in order to ensure the diversity, other contracts should also have a certain probability of being selected. Therefore, we use the dynamic priority scheduling algorithm to maintain a candidate queue. For each contract, we give it an initial priority, and then its value changes with the increasing of waiting time to ensure that every seed has the chance to be selected.

As Algo. 1 shows, the priority of each seed contract consists of two parts (Algorithm 1 line 3-4). The first part is metric difference priority, and the initial value is a number between 0 and 10, which is proportional to the value of difference; the second part is time priority, and the initial value is 0. Then, all candidate seed contracts are sorted according to the priority value, and the contract with the highest integrated priority is selected as the next mutation object, and the time priority of other seed contracts is increased for next iteration (Algorithm 1 line 9-11).