Arachne: Search Based Repair of Deep Neural Networks

TL;DR

Arachne is a novel DNN repair method that directly fixes misbehaviours using input-output pairs, localising weights and optimizing them with Differential Evolution, achieving effective repairs with high generalisation and addressing fairness issues.

Contribution

Introduces Arachne, a new program repair technique for DNNs that localises and optimizes weights to fix misclassifications efficiently and effectively.

Findings

Arachne can fix specific misclassifications without significantly reducing overall accuracy.

Patches by Arachne generalise to 61.3% of unseen misbehaviour, outperforming state-of-the-art methods.

Arachne can address fairness issues and generalise beyond CNNs.

Abstract

The rapid and widespread adoption of Deep Neural Networks (DNNs) has called for ways to test their behaviour, and many testing approaches have successfully revealed misbehaviour of DNNs. However, it is relatively unclear what one can do to correct such behaviour after revelation, as retraining involves costly data collection and does not guarantee to fix the underlying issue. This paper introduces Arachne, a novel program repair technique for DNNs, which directly repairs DNNs using their input-output pairs as a specification. Arachne localises neural weights on which it can generate effective patches and uses Differential Evolution to optimise the localised weights and correct the misbehaviour. An empirical study using different benchmarks shows that Arachne can fix specific misclassifications of a DNN without reducing general accuracy significantly. On average, patches generated by Arachne generalise to 61.3% of unseen misbehaviour, whereas those by a state-of-the-art DNN repair technique generalise only to 10.2% and sometimes to none while taking tens of times more than Arachne. We also show that Arachne can address fairness issues by debiasing a gender classification model. Finally, we successfully apply Arachne to a text sentiment model to show that it generalises beyond Convolutional Neural Networks.