Benchmark Assessment for DeepSpeed Optimization Library

TL;DR

This paper evaluates the Microsoft DeepSpeed library's impact on training efficiency across various neural network architectures, revealing mixed results with some improvements and some negative effects.

Contribution

It extends prior assessments of DeepSpeed by evaluating its performance on modern neural networks like CNNs and ViT, providing a broader understanding of its effectiveness.

Findings

DeepSpeed improves training efficiency in some architectures.

In certain cases, DeepSpeed has no effect or negative impact.

Evaluation covers modern neural network architectures beyond LeNet.

Abstract

Deep Learning (DL) models are widely used in machine learning due to their performance and ability to deal with large datasets while producing high accuracy and performance metrics. The size of such datasets and the complexity of DL models cause such models to be complex, consuming large amount of resources and time to train. Many recent libraries and applications are introduced to deal with DL complexity and efficiency issues. In this paper, we evaluated one example, Microsoft DeepSpeed library through classification tasks. DeepSpeed public sources reported classification performance metrics on the LeNet architecture. We extended this through evaluating the library on several modern neural network architectures, including convolutional neural networks (CNNs) and Vision Transformer (ViT). Results indicated that DeepSpeed, while can make improvements in some of those cases, it has no or negative impact on others.