Batch Size Influence on Performance of Graphic and Tensor Processing Units during Training and Inference Phases

TL;DR

This study investigates how batch size affects GPU and TPU performance during training and inference, demonstrating significant speedups with larger batch sizes on standard datasets without compromising accuracy.

Contribution

It provides empirical evidence of performance gains using large batch sizes on TPUs compared to GPUs, with detailed speedup metrics for training and inference phases.

Findings

Speedup up to 10x for training on TPUv2 compared to GPU

Speedup up to 2x for inference with TPUv2

No loss in prediction accuracy across tested batch sizes

Abstract

The impact of the maximally possible batch size (for the better runtime) on performance of graphic processing units (GPU) and tensor processing units (TPU) during training and inference phases is investigated. The numerous runs of the selected deep neural network (DNN) were performed on the standard MNIST and Fashion-MNIST datasets. The significant speedup was obtained even for extremely low-scale usage of Google TPUv2 units (8 cores only) in comparison to the quite powerful GPU NVIDIA Tesla K80 card with the speedup up to 10x for training stage (without taking into account the overheads) and speedup up to 2x for prediction stage (with and without taking into account overheads). The precise speedup values depend on the utilization level of TPUv2 units and increase with the increase of the data volume under processing, but for the datasets used in this work (MNIST and Fashion-MNIST with images of sizes 28x28) the speedup was observed for batch sizes >512 images for training phase and >40 000 images for prediction phase. It should be noted that these results were obtained without detriment to the prediction accuracy and loss that were equal for both GPU and TPU runs up to the 3rd significant digit for MNIST dataset, and up to the 2nd significant digit for Fashion-MNIST dataset.