YFlows: Systematic Dataflow Exploration and Code Generation for Efficient Neural Network Inference using SIMD Architectures on CPUs

TL;DR

This paper introduces YFlows, a system that explores neural network dataflows and generates optimized SIMD code for CPU inference, significantly improving speed while maintaining accuracy.

Contribution

YFlows presents a novel dataflow analysis and code generation framework that optimizes neural network inference on CPUs using SIMD architectures.

Findings

Up to 3x speedup for 8-bit neural networks.

Up to 4.8x speedup for binary neural networks.

Dataflow strategies that maximize reuse yield best performance.

Abstract

We address the challenges associated with deploying neural networks on CPUs, with a particular focus on minimizing inference time while maintaining accuracy. Our novel approach is to use the dataflow (i.e., computation order) of a neural network to explore data reuse opportunities using heuristic-guided analysis and a code generation framework, which enables exploration of various Single Instruction, Multiple Data (SIMD) implementations to achieve optimized neural network execution. Our results demonstrate that the dataflow that keeps outputs in SIMD registers while also maximizing both input and weight reuse consistently yields the best performance for a wide variety of inference workloads, achieving up to 3x speedup for 8-bit neural networks, and up to 4.8x speedup for binary neural networks, respectively, over the optimized implementations of neural networks today.