A Comprehensive Study of Deep Learning Model Fixing Approaches

TL;DR

This paper conducts a large-scale empirical evaluation of 16 deep learning model fixing approaches across various levels, assessing their effectiveness and impact on properties like robustness and fairness, providing insights for future research.

Contribution

It offers a comprehensive empirical comparison of state-of-the-art DL fixing methods, evaluating multiple properties and datasets to guide future research and industry practices.

Findings

Model-level approaches have the highest fixing effectiveness.

No single approach excels in all properties.

Mitigating side effects remains a key research challenge.

Abstract

Deep Learning (DL) has been widely adopted in diverse industrial domains, including autonomous driving, intelligent healthcare, and aided programming. Like traditional software, DL systems are also prone to faults, whose malfunctioning may expose users to significant risks. Consequently, numerous approaches have been proposed to address these issues. In this paper, we conduct a large-scale empirical study on 16 state-of-the-art DL model fixing approaches, spanning model-level, layer-level, and neuron-level categories, to comprehensively evaluate their performance. We assess not only their fixing effectiveness (their primary purpose) but also their impact on other critical properties, such as robustness, fairness, and backward compatibility. To ensure comprehensive and fair evaluation, we employ a diverse set of datasets, model architectures, and application domains within a uniform experimental setup for experimentation. We summarize several key findings with implications for both industry and academia. For example, model-level approaches demonstrate superior fixing effectiveness compared to others. No single approach can achieve the best fixing performance while improving accuracy and maintaining all other properties. Thus, academia should prioritize research on mitigating these side effects. These insights highlight promising directions for future exploration in this field.