PreNeT: Leveraging Computational Features to Predict Deep Neural Network Training Time

TL;DR

PreNeT is a predictive framework that accurately estimates deep neural network training times across different hardware setups by analyzing computational features, helping optimize training configurations and reduce costs.

Contribution

It introduces PreNeT, a novel method integrating detailed computational metrics to improve training time prediction accuracy for diverse hardware architectures.

Findings

Achieves up to 72% improvement in prediction accuracy

Effectively predicts training duration on new hardware architectures

Enhances existing prediction methods with detailed layer analysis

Abstract

Training deep learning models, particularly Transformer-based architectures such as Large Language Models (LLMs), demands substantial computational resources and extended training periods. While optimal configuration and infrastructure selection can significantly reduce associated costs, this optimization requires preliminary analysis tools. This paper introduces PreNeT, a novel predictive framework designed to address this optimization challenge. PreNeT facilitates training optimization by integrating comprehensive computational metrics, including layer-specific parameters, arithmetic operations and memory utilization. A key feature of PreNeT is its capacity to accurately predict training duration on previously unexamined hardware infrastructures, including novel accelerator architectures. This framework employs a sophisticated approach to capture and analyze the distinct characteristics of various neural network layers, thereby enhancing existing prediction methodologies. Through proactive implementation of PreNeT, researchers and practitioners can determine optimal configurations, parameter settings, and hardware specifications to maximize cost-efficiency and minimize training duration. Experimental results demonstrate that PreNeT achieves up to 72% improvement in prediction accuracy compared to contemporary state-of-the-art frameworks.