Hardware Acceleration for Neural Networks: A Comprehensive Survey

TL;DR

This survey comprehensively reviews hardware acceleration techniques for neural networks, covering architectures, optimization strategies, and open challenges across various workloads and deployment settings.

Contribution

It provides a unified taxonomy and synthesis of current hardware architectures, software tools, and emerging trends in neural network acceleration.

Findings

Highlights key architectural innovations like systolic arrays and specialized kernels.

Identifies open challenges such as efficient long-context LLM inference and sparse workload support.

Discusses future directions including energy efficiency and fair benchmarking.

Abstract

Neural networks have become dominant computational workloads across cloud and edge platforms, but their rapid growth in model size and deployment diversity has exposed hardware bottlenecks increasingly dominated by memory movement, communication, and irregular operators rather than peak arithmetic throughput. This survey reviews the current technology landscape for hardware acceleration of deep learning, spanning GPUs and tensor-core architectures, domain-specific accelerators (TPUs, NPUs), FPGA-based designs, ASIC inference engines, and emerging LLM-serving accelerators such as LPUs, alongside in-/near-memory computing and neuromorphic/analog approaches. We organize the survey using a unified taxonomy across (i) workloads (CNNs, RNNs, GNNs, Transformers/LLMs), (ii) execution settings (training vs.\ inference; datacenter vs.\ edge), and (iii) optimization levers (reduced precision, sparsity and pruning, operator fusion, compilation and scheduling, memory-system/interconnect design). We synthesize key architectural ideas such as systolic arrays, vector and SIMD engines, specialized attention and softmax kernels, quantization-aware datapaths, and high-bandwidth memory, and discuss how software stacks and compilers bridge model semantics to hardware. Finally, we highlight open challenges -- including efficient long-context LLM inference (KV-cache management), robust support for dynamic and sparse workloads, energy- and security-aware deployment, and fair benchmarking -- pointing to promising directions for the next generation of neural acceleration.