DEEP-GAP: Deep-learning Evaluation of Execution Parallelism in GPU Architectural Performance

TL;DR

DEEP-GAP systematically evaluates GPU inference performance across T4 and L4 architectures, revealing how precision modes and batch sizes impact throughput and efficiency for modern AI workloads.

Contribution

This work extends the GDEV-AI methodology to GPU inference, providing empirical data comparing T4 and L4 GPUs under controlled conditions.

Findings

INT8 precision achieves up to 58x throughput over CPU baseline.

L4 GPU outperforms T4 by up to 4.4x in throughput.

Smaller batch sizes favor L4 for latency-sensitive tasks.

Abstract

Modern datacenters increasingly rely on low-power, single-slot inference accelerators to balance performance, energy efficiency, and rack density constraints. The NVIDIA T4 GPU has become widely deployed due to strong performance per watt and mature software support. Its successor, the NVIDIA L4 GPU, introduces improvements in Tensor Core throughput, cache capacity, memory bandwidth, and parallel execution capability. However, limited empirical evidence quantifies the practical inference performance gap between these two generations under controlled and reproducible conditions. This work introduces DEEP-GAP, a systematic evaluation extending the GDEV-AI methodology to GPU inference. Using identical configurations and workloads, we evaluate ResNet18, ResNet50, and ResNet101 across FP32, FP16, and INT8 precision modes using PyTorch and TensorRT. Results show that reduced precision significantly improves performance, with INT8 achieving up to 58x throughput improvement over CPU baselines. L4 achieves up to 4.4x higher throughput than T4 while reaching peak efficiency at smaller batch sizes between 16 and 32, improving latency-throughput tradeoffs for latency-sensitive workloads. T4 remains competitive for large batch workloads where cost or power efficiency is important. DEEP-GAP provides practical guidance for selecting precision modes, batch sizes, and GPU architectures for modern inference deployments.