Fault Localisation and Repair for DL Systems: An Empirical Study with LLMs

TL;DR

This study evaluates existing fault localisation and repair techniques for deep learning models, introduces a novel LLM-based approach, and demonstrates GPT-4's significant improvements in these tasks on a new benchmark.

Contribution

It provides a comprehensive evaluation of current methods and introduces a novel LLM-based approach that significantly enhances fault localisation and repair in DL systems.

Findings

GPT-4 achieves 44% improvement in fault localisation

GPT-4 achieves 82% improvement in repair tasks

Current techniques have notable limitations in accuracy

Abstract

Numerous Fault Localisation (FL) and repair techniques have been proposed to address faults in Deep Learning (DL) models. However, their effectiveness in practical applications remains uncertain due to the reliance on pre-defined rules. This paper presents a comprehensive evaluation of state-of-the-art FL and repair techniques, examining their advantages and limitations. Moreover, we introduce a novel approach that harnesses the power of Large Language Models (LLMs) in localising and repairing DL faults. Our evaluation, conducted on a carefully designed benchmark, reveals the strengths and weaknesses of current FL and repair techniques. We emphasise the importance of enhanced accuracy and the need for more rigorous assessment methods that employ multiple ground truth patches. Notably, LLMs exhibit remarkable performance in both FL and repair tasks. For instance, the GPT-4 model achieves 44% and 82% improvements in FL and repair tasks respectively, compared to the second-best tool, demonstrating the potential of LLMs in this domain. Our study sheds light on the current state of FL and repair techniques and suggests that LLMs could be a promising avenue for future advancements.