An Effective Data-Driven Approach for Localizing Deep Learning Faults

TL;DR

This paper introduces a novel data-driven method that uses model features to automatically detect and diagnose faults in deep learning models, improving debugging efficiency and accuracy.

Contribution

It presents a new approach leveraging semantic features for fault detection in DNNs, automatically linking symptoms to root causes without manual mappings.

Findings

Effective detection of various bug types in real-world and mutated models.

Outperforms prior work in accuracy, precision, and recall on mutated models.

Achieves comparable results to state-of-the-art on real-world models.

Abstract

Deep Learning (DL) applications are being used to solve problems in critical domains (e.g., autonomous driving or medical diagnosis systems). Thus, developers need to debug their systems to ensure that the expected behavior is delivered. However, it is hard and expensive to debug DNNs. When the failure symptoms or unsatisfied accuracies are reported after training, we lose the traceability as to which part of the DNN program is responsible for the failure. Even worse, sometimes, a deep learning program has different types of bugs. To address the challenges of debugging DNN models, we propose a novel data-driven approach that leverages model features to learn problem patterns. Our approach extracts these features, which represent semantic information of faults during DNN training. Our technique uses these features as a training dataset to learn and infer DNN fault patterns. Also, our methodology automatically links bug symptoms to their root causes, without the need for manually crafted mappings, so that developers can take the necessary steps to fix faults. We evaluate our approach using real-world and mutated models. Our results demonstrate that our technique can effectively detect and diagnose different bug types. Finally, our technique achieved better accuracy, precision, and recall than prior work for mutated models. Also, our approach achieved comparable results for real-world models in terms of accuracy and performance to the state-of-the-art.