Imitation Game: Reproducing Deep Learning Bugs Leveraging an Intelligent Agent

TL;DR

This paper introduces RepGen, an automated intelligent system that significantly improves the reproduction of deep learning bugs, addressing the challenge of nondeterminism and environmental coupling in DL applications.

Contribution

RepGen is a novel approach that leverages learning-enhanced context and an iterative generate-validate-refine process with an LLM to reproduce DL bugs more effectively.

Findings

Achieves 80.19% bug reproduction rate, outperforming previous methods.

Reduces bug reproduction time by 56.8%.

Decreases cognitive load for developers.

Abstract

Despite their wide adoption in various domains (e.g., healthcare, finance, software engineering), Deep Learning (DL)-based applications suffer from many bugs, failures, and vulnerabilities. Reproducing these bugs is essential for their resolution, but it is extremely challenging due to the inherent nondeterminism of DL models and their tight coupling with hardware and software environments. According to recent studies, only about 3% of DL bugs can be reliably reproduced using manual approaches. To address these challenges, we present RepGen, a novel, automated, and intelligent approach for reproducing deep learning bugs. RepGen constructs a learning-enhanced context from a project, develops a comprehensive plan for bug reproduction, employs an iterative generate-validate-refine mechanism, and thus generates such code using an LLM that reproduces the bug at hand. We evaluate RepGen on 106 real-world deep learning bugs and achieve a reproduction rate of 80.19%, a 19.81% improvement over the state-of-the-art measure. A developer study involving 27 participants shows that RepGen improves the success rate of DL bug reproduction by 23.35%, reduces the time to reproduce by 56.8%, and lowers participants' cognitive load.