Repairing DNN Architecture: Are We There Yet?

TL;DR

This paper surveys and evaluates current techniques for repairing DNN models, revealing that existing methods often underperform on complex models and highlighting the need for more advanced repair strategies.

Contribution

It provides a comprehensive evaluation of state-of-the-art DNN repair techniques using real-world and artificial faults, exposing their limitations on larger models.

Findings

Random baseline performs comparably or better than existing techniques.

All repair techniques struggle with larger, complex models.

Current methods often fail to find fixes for sophisticated models.

Abstract

As Deep Neural Networks (DNNs) are rapidly being adopted within large software systems, software developers are increasingly required to design, train, and deploy such models into the systems they develop. Consequently, testing and improving the robustness of these models have received a lot of attention lately. However, relatively little effort has been made to address the difficulties developers experience when designing and training such models: if the evaluation of a model shows poor performance after the initial training, what should the developer change? We survey and evaluate existing state-of-the-art techniques that can be used to repair model performance, using a benchmark of both real-world mistakes developers made while designing DNN models and artificial faulty models generated by mutating the model code. The empirical evaluation shows that random baseline is comparable with or sometimes outperforms existing state-of-the-art techniques. However, for larger and more complicated models, all repair techniques fail to find fixes. Our findings call for further research to develop more sophisticated techniques for Deep Learning repair.