FPRaker: A Processing Element For Accelerating Neural Network Training

TL;DR

FPRaker is a novel processing element designed to accelerate neural network training by efficiently handling multiply-accumulate operations, exploiting natural data sparsity, and supporting various training optimizations for improved performance and energy efficiency.

Contribution

It introduces FPRaker, a processing element that processes multiply-accumulate operations in a power-of-two format, enabling skipping of ineffectual work and enhancing training acceleration.

Findings

FPRaker improves training performance and energy efficiency over conventional units.

It benefits training with pruning and quantization.

It enhances performance with layer-wise precision adjustments.

Abstract

We present FPRaker, a processing element for composing training accelerators. FPRaker processes several floating-point multiply-accumulation operations concurrently and accumulates their result into a higher precision accumulator. FPRaker boosts performance and energy efficiency during training by taking advantage of the values that naturally appear during training. Specifically, it processes the significand of the operands of each multiply-accumulate as a series of signed powers of two. The conversion to this form is done on-the-fly. This exposes ineffectual work that can be skipped: values when encoded have few terms and some of them can be discarded as they would fall outside the range of the accumulator given the limited precision of floating-point. We demonstrate that FPRaker can be used to compose an accelerator for training and that it can improve performance and energy efficiency compared to using conventional floating-point units under ISO-compute area constraints. We also demonstrate that FPRaker delivers additional benefits when training incorporates pruning and quantization. Finally, we show that FPRaker naturally amplifies performance with training methods that use a different precision per layer.