Do Automated Fixes Truly Mitigate Smart Contract Exploits?

TL;DR

This paper evaluates the effectiveness of automated program repair tools in mitigating exploits in smart contracts, revealing significant disparities and systemic limitations in current approaches.

Contribution

Introduces a systematic framework and the first measure of exploit mitigation rate for smart contract repair tools, providing new insights into their real-world effectiveness.

Findings

Mitigation rates range from 29% to 74%.

Substantial disparities exist among state-of-the-art tools.

Identifies systemic limitations like inconsistent functionality preservation.

Abstract

Automated Program Repair (APR) for smart contract security promises to automatically mitigate smart contract vulnerabilities responsible for billions in financial losses. However, the true effectiveness of this research in addressing smart contract exploits remains uncharted territory. This paper bridges this critical gap by introducing a novel and systematic experimental framework for evaluating exploit mitigation of program repair tools for smart contracts. We qualitatively and quantitatively analyze 20 state-of-the-art APR tools using a dataset of 143 vulnerable smart contracts, for which we manually craft 91 executable exploits. We are the very first to define and measure the essential "exploit mitigation rate" , giving researchers and practitioners a real sense of effectiveness of cutting edge techniques. Our findings reveal substantial disparities in the state of the art, with an exploit mitigation rate ranging from a low of 29% to a high of 74%. Our study identifies systemic limitations, such as inconsistent functionality preservation, that must be addressed in future research on program repair for smart contracts.