Towards Training Reproducible Deep Learning Models

TL;DR

This paper presents a systematic approach to enhance the reproducibility of deep learning models by addressing software randomness and hardware non-determinism, validated through case studies on multiple models.

Contribution

It introduces a comprehensive framework combining record-and-replay and profile-and-patch techniques, along with evaluation criteria and guidelines for reproducible deep learning training.

Findings

Successfully reproduces six open-source DL models

Effectively mitigates software and hardware sources of randomness

Provides a practical guideline for reproducible DL training

Abstract

Reproducibility is an increasing concern in Artificial Intelligence (AI), particularly in the area of Deep Learning (DL). Being able to reproduce DL models is crucial for AI-based systems, as it is closely tied to various tasks like training, testing, debugging, and auditing. However, DL models are challenging to be reproduced due to issues like randomness in the software (e.g., DL algorithms) and non-determinism in the hardware (e.g., GPU). There are various practices to mitigate some of the aforementioned issues. However, many of them are either too intrusive or can only work for a specific usage context. In this paper, we propose a systematic approach to training reproducible DL models. Our approach includes three main parts: (1) a set of general criteria to thoroughly evaluate the reproducibility of DL models for two different domains, (2) a unified framework which leverages a record-and-replay technique to mitigate software-related randomness and a profile-and-patch technique to control hardware-related non-determinism, and (3) a reproducibility guideline which explains the rationales and the mitigation strategies on conducting a reproducible training process for DL models. Case study results show our approach can successfully reproduce six open source and one commercial DL models.