ConvBench: A Comprehensive Benchmark for 2D Convolution Primitive Evaluation

TL;DR

ConvBench is a detailed benchmark tool that evaluates 2D convolution algorithms at the primitive level, enabling fair comparison and revealing optimization opportunities, demonstrated through analysis of the Sliced Convolution algorithm.

Contribution

This paper introduces ConvBench, a comprehensive primitive-level benchmark for 2D convolution algorithms, addressing limitations of previous ad-hoc testing suites and enabling detailed performance analysis.

Findings

ConvBench evaluated 9243 convolution operations from 1097 models.

Sliced Convolution outperformed Im2col-GEMM in 93.6% of cases.

Identified a 79.5% slowdown in SConv's packing step, suggesting optimization potential.

Abstract

Convolution is a compute-intensive operation placed at the heart of Convolution Neural Networks (CNNs). It has led to the development of many high-performance algorithms, such as Im2col-GEMM, Winograd, and Direct-Convolution. However, the comparison of different convolution algorithms is an error-prone task as it requires specific data layouts and system resources. Failure to address these requirements might lead to unwanted time penalties. Thus, considering all processing steps within convolution algorithms is essential to comprehensively evaluate and fairly compare their performance. Furthermore, most known convolution benchmarking adopts ad-hoc testing suites with limited coverage and handmade operations. This paper proposes ConvBench, a primitive-level benchmark for the evaluation and comparison of convolution algorithms. It assesses 9243 convolution operations derived from 1097 real-world deep learning models, resulting in performance and execution breakdown graphs for a detailed evaluation. ConvBench capability is evaluated across the Sliced Convolution (SConv) algorithm. The experiments showed results faster than Im2col-GEMM in 93.6% of the convolutions. However, the use of ConvBench allowed the delving into the remaining 6.4% underperforming convolutions, uncovering a critical slowdown of 79.5% on average of SConv's packing step. This analysis underscores a potential source of optimization for SConv, opening up new paths for convolution designers to improve their algorithms.