Mutation testing of smart contracts at scale

TL;DR

This paper evaluates the effectiveness of mutation testing for smart contracts at scale, analyzing existing operators, identifying vulnerabilities, and introducing a new method to detect deviations in gas consumption.

Contribution

It assesses mutation testing effectiveness on large-scale smart contracts and proposes a novel gas consumption deviation detection method.

Findings

Existing mutation operators have limited killability.

Severe vulnerabilities can be injected via mutations.

The new gas deviation detection improves mutation testing effectiveness.

Abstract

It is crucial that smart contracts are tested thoroughly due to their immutable nature. Even small bugs in smart contracts can lead to huge monetary losses. However, testing is not enough; it is also important to ensure the quality and completeness of the tests. There are already several approaches that tackle this challenge with mutation testing, but their effectiveness is questionable since they only considered small contract samples. Hence, we evaluate the quality of smart contract mutation testing at scale. We choose the most promising of the existing (smart contract specific) mutation operators, analyse their effectiveness in terms of killability and highlight severe vulnerabilities that can be injected with the mutations. Moreover, we improve the existing mutation methods by introducing a novel killing condition that is able to detect a deviation in the gas consumption, i.e., in the monetary value that is required to perform transactions. This paper has a replication package at https://github.com/pieterhartel/Mutation-at-scale