Implementation of high-performance, sub-microsecond deep neural networks on FPGAs for trigger applications

TL;DR

This paper presents a specialized FPGA framework enabling high-performance, low-latency deep neural network processing for trigger systems in physics experiments, achieving near-maximal efficiency and minimal latency.

Contribution

It introduces a novel FPGA-based implementation framework for deep neural networks tailored to trigger applications, optimizing layer design for high efficiency and low latency.

Findings

Achieves 90-100% processing efficiency for large neural network layers.

Latencies are reduced to tens to hundreds of nanoseconds for entire networks.

Framework supports efficient creation of FPGA implementations from trained models.

Abstract

Artificial neural networks are already widely used for physics analysis, but there are only few applications within low-level hardware triggers, and typically only with small networks. Modern high-end FPGAs offer Tera-scale arithmetic performance, and thereby provide a significant amount of operations per data set even for MHz-range data rates. We present a bottom-up approach of implementing typical neural network layers, in which we took both the special constraints that come from high-performance trigger systems, such as the ATLAS hardware trigger at the LHC, as well as an efficient implementation into account. By specifically designing each layer type to match our requirements, we could develop a framework that reaches 90 to 100% processing efficiency for large layers, requires only few extra resources for data flow and controlling, and offers latencies in the range of only tens to hundreds of nanoseconds for entire (deep) networks. Additionally, a toolkit was built around these optimized layer implementations, which facilitates the creation of the FPGA implementation of a trained NN model.

Figures

Tables

Table 1. Table 1 : Implementation results for example networks trained for the MNIST digit recognition task. Activation relu used for all but the last layer, which had a ’linear’ activation and always had N N = 10 subscript 𝑁 N 10 N_{\mathrm{N}}=10 neurons. The single-channel input has a size of 14 × 14 14 14 14\times 14 , if not stated otherwise. WNS was left out where no WNS occurred, i.e. timing was met for these designs. For convolutional layers, the kernel shape ( H K × W K × N K ) subscript 𝐻 K subscript 𝑊 K subscript 𝑁 K (H_{\mathrm{K}}\times W_{\mathrm{K}}\times N_{\mathrm{K}}) is specified, for pooling layers the pooling area ( H P × W P ) subscript 𝐻 P subscript 𝑊 P (H_{\mathrm{P}}\times W_{\mathrm{P}}) and for fully-connected layers the number of neurons N N subscript 𝑁 N N_{\mathrm{N}} is specified. The DSP efficiency refers to the relative amount of non-idling DSP cycles. For reference, the Xilinx UltraScale+ XCVU9P target FPGA features 6840 DSPs, 2160 BRAMs, approximately 1.2 million LUTs and twice as many FFs.

Architecture (see text)	MACs	$T_P$	WNS	latency	$N_{\mathrm{LUT}}$	$N_{\mathrm{FF}}$
(layer information)	(DSP eff.)	(ns)	(ns)	(cycles)	$N_{\mathrm{DSP}}$	$N_{\mathrm{BRAM}}$
ArcA1 ($C=16$) (input $\left(7\times 7\right)$)	334	1.562	-	56	1793	3571
$\left(2\times 2\times 1\right)$-$\left(2\times 2\right)$-$10$	(0.485)				43	10.5
\arrayrulecolorgray \cdashline1-7 ArcA2 ($C=14$)	1089	1.786	-	60	5060	9706
$\left(2\times 2\times 1\right)$-$\left(2\times 2\right)$-$7$	(0.630)				108	17
\cdashline1-7 ArcA3 ($C=14$) (input $\left(7\times 7\right)$)	1024	1.786	-	57	3051	5654
$\left(2\times 2\times 3\right)$-$\left(2\times 2\right)$-$16)$	(0.620)				118	19
\cdashline1-7 ArcA4 ($C=13$)	3188	1.923	-	63	8689	16219
$\left(2\times 2\times 2\right)$-$\left(2\times 2\right)$-$17$)	(0.774)				317	54.5
\cdashline1-7 ArcA5 ($C=13$)	7854	1.923	-	68	15567	28450
$\left(2\times 2\times 4\right)$-$\left(2\times 2\right)$-$25$	(0.967)				625	93.5
\cdashline1-7 ArcA6 ($C=11$)	12884	2.273	-	68	20962	34711
$\left(3\times 3\times 4\right)$-$\left(2\times 2\right)$-$50$	(0.894)				1310	166
\cdashline1-7 ArcB1 ($C=12$)	8858	2.083	-	76	18587	32886
$\left(2\times 2\times 4\right)$-$\left(2\times 2\right)$-$\left(2\times 2\times 4\right)$-$25$	(0.812)				909	99.5
\cdashline1-7 ArcB1 ($C=16$)	8858	2.083	-	87	17205	32760
$\left(2\times 2\times 4\right)$-$\left(2\times 2\right)$-$\left(2\times 2\times 4\right)$-$25$	(0.812)				713	71.5
\cdashline1-7 ArcB3 ($C=11$)	11362	2.273	-	79	28383	47140
$\left(2\times 2\times 6\right)$-$\left(2\times 2\right)$-$\left(2\times 2\times 4\right)$-$25$	(0.792)				1305	102.5
\arrayrulecolorblack ArcB2 ($C=10$)			15610	2.500	-0.134	84	40998	69333
$\left(3\times 3\times 6\right)$-$\left(2\times 2\right)$-$\left(3\times 3\times 6\right)$-25	(0.855)				1825	68
\arrayrulecolorgray \cdashline1-7 ArcB3 ($C=16$)	11362	1.562	-0.014	93	26006	45065
$\left(2\times 2\times 6\right)$-$\left(2\times 2\right)$-$\left(2\times 2\times 4\right)$-25	(0.825)				861	71.5
\cdashline1-7 ArcC1 ($C=8$)	24076	3.125	-0.045	93	37528	61388
$\left(3\times 3\times 6\right)$-$\left(2\times 2\right)$-$\left(2\times 2\times 8\right)$-50-25	(0.934)				3222	338.5
\cdashline1-7 ArcD5 ($C=9$)	26120	2.778	-0.060	86	32592	51865
$\left(2\times 2\times 4\right)-\left(2\times 2\times 2\right)-\left(2\times 2\times 2\right)-50-25$	(0.928)				3128	353

Equations

$$\vec{o}=\vec{A}\left(W\cdot \vec{i}+\vec{b}\right)$$

Full text

\keywords

artificial neural networks, deep neural networks, neural network inference, FPGA, detector trigger, hardware trigger, sub-microsecond latency

1 Introduction

Deep neural networks (DNNs) are already used in a wide range of inference tasks, such as speech and image recognition, and are continuously advancing into physics, until now mostly for offline data analysis. There are only very few examples where DNNs are used in or targeted for the context of high-performance detector triggers. This might be due to the very special inference rate and latency constraints that apply in such environments, and the difficulty of developing complex algorithms for field-programmable gate arrays (FPGAs). FPGAs, however, often are the only type of processing hardware that can be used in such contexts, apart from even more demanding ASICs. One example of a specific network in the trigger context can be found in the Belle II trigger, in which a relatively small neural network is used for z-vertex triggering [1]. An example for a more general attempt at enabling neural network usage within triggers is the ”High Level Synthesis for Machine Learning” (hls4ml) companion compiler, which uses high-level synthesis tools to generate the FPGA firmware design for a given network [2]. A more detailed overview on trigger requirements and existing work regarding neural network inference on FPGAs is also given in [2]. In this paper we take the ATLAS detector [3] and its upgrades of the first level trigger system [4, 5] as a reference for our studies. In this FPGA-based trigger level, the incoming data rate of $40\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$ needs to be reduced down to less than $100\text{\,}\mathrm{k}\mathrm{H}\mathrm{z}$ within a maximal latency of $2.2\text{\,}\mathrm{\SIUnitSymbolMicro s}$. Only this large reduction in rate allows for the further processing by a software trigger that reduces the event rate further to about $1\text{\,}\mathrm{k}\mathrm{H}\mathrm{z}$, the maximum rate that can be written to permanent storage. Most of the $2.2\text{\,}\mathrm{\SIUnitSymbolMicro s}$ latency is used up by data preparation and transfer, such that only a few tens to few hundreds of nanoseconds remain for actual neural network applications.

In contrast to the hls4ml framework, we chose to pursue a hardware-centric, bottom-up approach for implementing general neural networks on FPGAs, which grants maximum control over the FPGA design, and therefore allows very fine tuning for the specific use case. Thereby, we intend to further lift network size limits, while simultaneously providing scientists working on trigger algorithms with an easy-to-use tool for efficiently incorporating neural networks of sizes that were never used before into their systems.

We begin with a very brief overview over the relevant aspects of neural networks as well as FPGAs. Following that, we discuss the required types of operations, how they can be implemented in hardware, and introduce the concept of user-configurable fixed-point precision. The main section of this paper focusses on the design of the individual layers, which currently include fully-connected layers, as well as two-dimensional (multi-channeled) convolutions and maximum pooling, and concludes this with the implementation of activation functions. We then quickly discuss what needs to be considered for putting multiple layers together in a functioning network, before the presentation of implementation results on individual layers and entire networks. In the end, we summarize the results of our developments and give an outlook on possible future improvements.

2 Basics

2.1 Neural networks

In the following, we are focussing on deep neural networks which consist of fully-connected, 2D convolutional and 2D maximum pooling layers, and any meaningful combination of these, i.e. an arbitrary combination of the 2D layers, which might be followed by flattening and then an arbitrary sequence of fully-connected layers, or alternatively a network consisting of fully-connected layers only. As neural network framework, Keras was used, with a TensorFlow backend [6].

In a fully-connected layer, every neuron receives every input, with the number of inputs $N_{\mathrm{I}}$ being predetermined by the network and the number of neurons $N_{\mathrm{N}}$ being a parameter of the layer. The inputs $\vec{i}\in\mathbb{R}^{N_{\mathrm{I}}}$ are multiplied by a weight matrix $W\in\mathbb{R}^{N_{\mathrm{N}}\times N_{\mathrm{I}}}$, typically have an offset $\vec{b}\in\mathbb{R}^{N_{\mathrm{N}}}$ applied and then go through a usually component-wise real activation function $\vec{A}:\mathbb{R}^{N_{\mathrm{N}}}\rightarrow\mathbb{R}^{N_{\mathrm{N}}}$ to produce the result $\vec{o}\in\mathbb{R}^{N_{N}}$, i.e. the fully-connected layer implements equation 1.

[TABLE]

A 2D convolutional layer receives an input image/feature map of shape $H_{\mathrm{I}}\times W_{\mathrm{I}}\times D_{\mathrm{I}}$. The number of kernels $N_{\mathrm{K}}$ and the kernel area $H_{\mathrm{K}}\times W_{\mathrm{K}}$ are parameters, while the kernel depth $D_{\mathrm{K}}$ equals $D_{\mathrm{I}}$ to incorporate all input channels into the kernel application. For a general layer, there are two additional architectural parameters called padding and stride. The stride determines the step lengths between kernel applications, and padding determines how kernel applications at the edges are treated. Currently, we support the ’default’ stride of 1 in both directions and what is usually called padding ’valid’, i.e. no incomplete kernel applications at the edges. Accordingly, output height and width are computed as $H_{\mathrm{O}}=H_{\mathrm{I}}-(H_{\mathrm{K}}-1)$ and $W_{\mathrm{O}}=W_{\mathrm{I}}-(W_{\mathrm{K}}-1)$, respectively, with the output depth $D_{\mathrm{O}}=N_{\mathrm{K}}$. The application of a kernel is similar to the fully-connected neuron operation, where $\vec{i}$ contains the inputs that are covered at the given position and $W$ depends on the kernel that is used.

As the 2D convolutional layer, a 2D pooling layer also receives a potentially multi-channeled input image. Instead of a kernel area, a pooling area $H_{\mathrm{P}}\times W_{\mathrm{P}}$ is defined, which is applied channel by channel and selects the maximum value within the given input range. As for the convolutional layer, currently only the default stride is supported, which in this case is $(H_{\mathrm{P}},W_{\mathrm{P}})$ (i.e. non-overlapping pooling areas without free spaces in between). Padding options ’valid’, ’same’ and ’unchanged’ are supported (see section 3.4). In consequence, the output shape is at least \big{\lfloor}\frac{H_{\mathrm{I}}}{H_{\mathrm{P}}}\big{\rfloor}\times\big{\lfloor}\frac{W_{\mathrm{I}}}{W_{\mathrm{P}}}\big{\rfloor}\times D_{\mathrm{I}}, with possibly one more element in height and width depending on the padding option and shapes.

Generally, it is also possible to reshape the data between layers. Until now, we support flattening from 2D multi-channeled data to 1D, where inputs are rearranged in a C-like manner, i.e. an input at $(x,y,channel)$ will be put into position $x\cdot(W_{\mathrm{I}}\cdot D_{\mathrm{I}})+y\cdot D_{\mathrm{I}}+channel$ in 1D.

2.2 FPGAs

Modern FPGAs comprise a large number (up to the order of millions) of programmable look-up tables (LUTs) and typically one or two flip-flop registers (FFs) per LUT, with the routing between these also being progammable. In the following, any specifics apply to the Xilinx UltraScale+ (US+) FPGA architecture [7, 8, 9, 10]. However, most features are directly or similarly applicable to other recent device families from Xilinx and competitors such as Intel/Altera.

The LUTs typically implement binary combinational functions from few bits to 1- or 2-bit results, e.g. $f:\{0,1\}^{6}\rightarrow\{0,1\}^{2}$ in the US+ architecture.111True di-output LUTs are only supported for up to 5 inputs, otherwise constraints apply, see [9]. A flip-flop can store a single bit.

The combination of a large ensemble of programmable logic functions, registers for storage and synchronous processing and programmable routing makes it possible to implement even complex and large digital circuits within FPGAs.

In addition to these basic building blocks, there are also specialized embedded components such as block memories (BRAMs) and digital signal processors (DSPs). The BRAMs provide a high-density, high-capacity memory. One BRAM block consists of two $18\text{\,}\mathrm{k}\mathrm{i}\mathrm{b}$ parts, where each is addressed by 9 to 14 address bits, which results in port widths and depths ranging from 36 and 512 to 1 and 16384. The DSPs are ’simple’ ALUs (arithmetic logic units), which offer a wide range of operations. In the US+ architecture, the operation mode which is most important for NN applications is a $w\cdot i+b$ type operation, where $w$, $i$ and $b$ are binary numbers of up to 18, 27, and 48 bits, respectively. The number $b$ can be an external input or come from an internal result register, allowing to perform a multiply-accumulate operation in either a pipelined (if $b$ is external) or localized (if $b$ is internally accumulated) design.

Our target device was the Xilinx US+ XCVU9P, which features approximately $1.2\text{\,}\mathrm{M}$ LUTs and $2.4\text{\,}\mathrm{M}$ FFs, as well as 6840 DSPs and 2160 2x18 kib BRAM units, and supports maximum operation frequencies in the range from $640\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$ to $900\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$, depending on device speed grade and hardware component. The FPGA firmware was designed in VHDL using Vivado 2018.2 as IDE with default settings for synthesis and implementation strategy, except for the synthesis mode out of context, to make it possible to implement designs without any I/O connections. Apart from automatic inference of hardware components such as LUTs, FFs and DSPs from the VHDL code, they were also explicitly instantiated by design primitives for the DSPs and adders implemented in the general logic, which gave maximum control over the implementation.

Key metrics for the evaluation of an FPGA implementation are the amount of resources required and the design timing, i.e. if the target frequency is met or if the design cannot run at the targeted frequency due to time needed for signal propagation. Designs which miss timing cannot be used without optimization, but even in these cases, it is useful to have knowledge about the severity of the timing violation.

2.3 Arithmetics implementation

Precision requirements

Previous work [2] has demonstrated that for NN inference, reasonably low precision is sufficient, and neither floating-point arithmetics nor a large number of bits are necessary for optimal performance in many cases.222Which we also verified for our own test implementations. Using fixed-point arithmetics provides the benefit of a significantly reduced implementation complexity, which saves resources and makes the adjustment of the arithmetic precision to specific needs regarding value range and granularity significantly easier.

In the following, we define the precision specification ’$i.f$’, where $i$ and $f$ are the integer and fractional bits in a fixed-point representation, with the entire value being in the so called two’s complement representation. Accordingly, the resulting value range is $-2^{i-1}$ to $2^{i-1}-G$ with the granularity $G=2^{-f}$.

Implementation in hardware

For DNNs as described above, only a few basic operations are needed: Multiplication and addition are combined into the multiply-accumulate (MAC) operation, and for the pooling layers, the ’select maximum’ operation (MAX) on two values is required.

Since fixed-point arithmetics are sufficient, it is possible to use only a single DSP to implement the MAC operation that is required for computing weighted input sums.333This of course limits the number of bits for the inputs, but with up to 27 by 18 bit-wide multiplications in the UltraScale+ architecture, this limit is significantly above what is usually critical for neural network purposes. Thereby, it is possible to provide one input and weight to a DSP per cycle, and then the DSP can add the product either to the internally accumulated value or to an externally provided partial result. For reaching the maximum DSP frequency, it is necessary to use internal pipeline registers of the DSPs. This requires having internal registers enabled at three levels: input, product and after the accumulation. Due to this, a ’first DSP’ latency of three cycles arises. For any additional DSP in a pipeline, only one extra cycle is necessary, as even an externally provided extra term (e.g. the current partial result) can be inserted latency-free before the accumulation step.444An extra external input register is also possible and might be required for future frequency improvements, but it is possible to incorporate this with only one extra cycle for the entire DSP pipeline, instead of one cycle per DSP.

In some situations, it is necessary to compute sums of values that come out of multiple DSP pipelines. Adding $N$ values requires $N-1$ adders. By adding multiple values in a binary tree structure, where at each level, pairs of values are added and the result is sent to the next level, it is possible to keep the logic depth at the optimal value of \big{\lceil}\log_{2}N\big{\rceil}, which is desirable from a latency point of view. To guarantee optimal performance, these logic-based adders were instantiated on a design primitive level, forcing Vivado to use the dedicated carry logic and to place the corresponding primitives as close as possible.

For the pooling layers, it is necessary to find the maximum in a set of values. Similarly to the sum of multiple values, this can be done in a binary tree structure, to provide the lowest possible logic depth and resource utilization.555It might be possible to use an algorithm with formally lower logic depth, but this is not expected to scale to the hardware regarding utilization and timing. Instead of being added, each pair of values is processed by the MAX operation, to detect the larger and send it to the next level, until only one value remains.

For the maximum of and sum of set structures, we added configuration parameters to include register stages at the input, inter-operation and output levels, allowing to trade maximum frequency against register utilization and latency in terms of cycles.

3 Layer structure

3.1 General decisions

Due to the very strict real-time context of high-performance detector triggers, it is very important to optimize the design for minimum latency. At the same time, it is necessary to aim for an optimized inference performance, which strongly relates to an efficient resource usage. It is not expected that a solution could be found which optimizes all three metrics at once, especially if no further constraints exist for the network architectures. The selection of potentially suitable design approaches is tightly bound to the data rate $f_{\mathrm{D}}=T_{\mathrm{D}}^{-1}$ (with the data tick period $T_{\mathrm{D}}$), the achievable processing frequency $f_{\mathrm{P}}$ and the latency budget. Motivated by the ATLAS first level trigger, we chose to optimize the design for a data rate of $40\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$ and for a latency as short as few tens of nanoseconds for entire networks, but put performance and efficiency before a perfectly minimized latency.

Generally, if data is coming in at a rate $f_{D}$ and the frequency of the processing units (PUs), e.g. the DSPs, is $f_{\mathrm{P}}$, there are C=\big{\lfloor}\frac{f_{\mathrm{P}}}{f_{\mathrm{D}}}\big{\rfloor} cycles per PU and data set for real-time processing. Moreover, three general structures could be used that potentially reach perfect efficiency: Ideally, one would have all of the hardware resources available to process an entire data set within one data tick, and then proceed with the next set, which also results in the shortest latency. If that is not possible, one can either build a single pipeline of processing steps, where partial results of one pipeline stage propagate to the next stage during each data tick, or partition the hardware into $N_{\mathrm{P}}$ parts, and place the corresponding number of instances of the network processors, where each has $N_{\mathrm{P}}$ data ticks for processing, and inputs are multiplexed to a different instance at each tick. We have decided for a single-instance pipelined design, and excluded the others:

The ’single tick processing’ design can be ruled out because with e.g. $T_{\mathrm{D}}=$25\text{,}\mathrm{n}\mathrm{s}$$ for the ATLAS detector, it will not be possible to have large designs which process all of the data within one data tick, simply due to the time required for computation and signal propagation in device-spanning, deep logic structures.

The decision for either a single pipeline design or a multi-instance design is less trivial, as both have advantages: In the single pipeline design, every pipeline stage could potentially be adapted to a specific task, which would reduce overhead structures. A multi-instance design would require more general processing units instead (as these would need to perform different tasks/compute different layers while time progresses), and could be implemented for example as systolic arrays. At the same time, data propagation through the single pipeline design would be bound to the pipeline structure, while the processing units in the multi-instance design could potentially allow better adjustment of the latency depending on the exact design, although at the price of an increased complexity.

We took three further aspects into account for our decision: In order to avoid an increase in latency, we decided not to batch the input data, i.e. the batch size during inference is 1. Additionally, with the given maximum hardware processing frequency and targeted data frequency range, $C$ is in the order of $10^{1}$, and in a neural network, a layer can only begin computing when at least some of the required inputs are available.

All of this convinced us to choose the single pipeline design: It makes it easier to keep the efficiency high if no batching is used and $C$ is small, it allows having relatively small (although possibly not minimized) latencies and it avoids too much overhead structure for reconfigurability and data flow management.666Even with this design, it is possible to further reduce the latency at cost of computational efficiency by constraining the layers to use less than $C$ cycles. The loss in computational efficiency would typically be tolerable, as mostly small networks would benefit from this, while larger networks are rather throughput-bound than latency-bound anyway. For the pipelining approach, it also appears natural to have one pipeline stage per network layer, which is conceptually much less complex than making efficient use of a systolic array while simultaneously keeping the latency low. Layers were therefore designed to take the data within $C$ cycles after the first part of the data arrives, and also produce all results within a period of $C$ cycles, to maintain network synchronicity, with an arbitrary but fixed delay in between.

Moreover, with $N_{\mathrm{PU}}$ processing units placeable in a device, there are $N_{\mathrm{Op}}=C\cdot N_{\mathrm{PU}}$ operations possible per data set and operation type, which is useful to understand the ideal-case inference performance of any device and thereby the maximum network size. Figure 1 shows the number of DSP MACs possible depending on the device/DSP count and processing frequency, where the ATLAS data frequency of $40\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$ was assumed. This shows that already with one of the current high-end FPGA families, a network size of up to 280 kMACs could be possible even for such high data frequencies, if the maximum device frequency can be achieved and the computational efficiency is close to 100%.

3.2 Fully-connected layers

Generally, it is possible to use a single DSP to implement the weighted input sum of a neuron, as the DSPs support the MAC operation. However, the number of processing cycles is limited, therefore arbitrary amounts of inputs $N_{\mathrm{I}}$ could only be handled if multiple inputs are weighted in parallel in multiple DSPs, and even then (or if $N_{\mathrm{I}}<C$), such an approach can be computationally very inefficient.

Instead, we exploited that neurons in fully-connected layers require each of the inputs in order to be computed. By combining this with the inputs being made available within $C$ cycles, we implemented the weighted input sums using pipelines of DSPs.

A typical pipelined data flow is illustrated in figure 2(a), where for illustrational purposes we assumed that per processing cycle, only one further input becomes available. In this scheme, there are as many DSPs as inputs to the given pipeline. The input $i_{n}$ ($n\in\mathbb{N}_{0}$), which is available at cycle $n$, is stored in a register which is connected to the input of $\mathrm{DSP}_{n}$ at the subsequent cycle edge, which keeps this input until it is overwritten after $C$ cycles. With $n+1$ being the first cycle when $\mathrm{DSP}_{n}$ has the new input available in its register, weight $w_{m,n}$ is presented to $\mathrm{DSP}_{n}$ during cycle $n+1+m$, where $m\in\mathbb{N}_{0}$ denotes the neuron index. After each cycle, the partially accumulated weighted input sum is passed as third input to the next DSP in the pipeline, which adds the next weighted input for the corresponding neuron, until the final result comes out of the last DSP.

Multiple of these pipelines can be used when multiple inputs become available during each cycle. We define the number of pipelines as $P$. Each pipeline weights at least \big{\lfloor}\frac{N_{\mathrm{I}}}{P}\big{\rfloor} inputs, and the first $N_{\mathrm{I}}\bmod P$ pipelines need to weight one extra input, while the remaining pipelines are extended by a shift register for cycle-matching the partial results.

Apart from parallelizing the pipelines when multiple inputs become available per processing cycle, it is also possible to use multiple of such ’neuron units’ (NUs) if necessary, i.e. N_{\mathrm{NU}}=\big{\lceil}\frac{N_{\mathrm{N}}}{C}\big{\rceil}>1. In that case, each NU computes at least \big{\lfloor}\frac{N_{\mathrm{N}}}{N_{\mathrm{NU}}}\big{\rfloor} neuron results, and the first $N_{\mathrm{N}}\bmod N_{\mathrm{NU}}$ NUs compute one extra neuron. If \big{\lceil}\frac{N_{\mathrm{N}}}{N_{\mathrm{NU}}}\big{\rceil}<C, all neuron results are available even before $C-1$ extra cycles after the first result, which allows a reduction of the layer-to-layer latency.

Code structure

As for the following layers, the fully-connected layer consists of multiple design subunits, which are shown in figure 2(b). The main part of the fully-connected layer are the neuron units. These contain the DSP pipelines, which use the $w\cdot i+b$ type operation of the DSPs, with $w$ for weights, $i$ for the input values, and $b$ as partial result of the previous DSP. In the current implementation, the $b$ signal can be implemented either via dedicated connections between the DSPs or through the general fabric wiring. The latter case provides more placement flexibility, but worse timing characteristics. For the fabric routing, timing could be further improved by also activating the DSP input register for the $b$ signal, which was not yet implemented.777Minor adaptations in the layer structure and controlling are necessary for that. If multiple pipelines are used, a final adder is included to sum up all partial results to produce the actual weighted neuron input sum.

The input values for the DSPs are stored in a memory entity with external write control, therefore the external write sequence must match the expected input sequence. Input $i$ is weighted by DSP \big{\lfloor}\frac{i}{P}\big{\rfloor} in pipeline $i\bmod P$. Similarly, there are multiple weight memory blocks, realized with BRAM.888The choice for BRAM-based memory units fell because we considered these a spare and available resource, which makes it possible to save LUTs and registers. If necessary, it would be possible to switch to memory based on the general logic resources. Currently, there is one weight memory block per NU and pipeline stage, which spans all DSPs of that stage and NU. In the future, other architectures might be added, such as blocks per pipeline or per part of a pipeline, for frequency optimization for different layer architectures.

In the end the results are multicast, i.e. the single result signal of any given neuron unit is then connected to all output signals of the entire dense layer which are produced by that neuron unit during any cycle. A controller entity controls the selection of the weight memory and asserts signals indicating when a result is valid.

3.3 2D convolutional layers

Convolutional layers typically feature only few parameters, but a significant amount of the total MAC operations within a network, due to the manifold application of the same kernel. Obviously, it is possible to apply kernels at multiple positions in parallel, as there are no direct dependencies between different output positions.999There are partially shared inputs, however, and some very intelligent ways to reduce the computational cost, e.g. by making use of the convolution theorem to implement the convolution as multiplication, but these often work well only for larger layers than possible here and sometimes require preparations that introduce significant additional latency, therefore we did not focus on these. However, not exploiting the topology of convolutional layers in any way results in a significant and mostly avoidable logic resource consumption, because the same input values and weights need to be copied often and in many places in spite of their redundancy.

Therefore, we studied possibilities to save resources by spatially and temporally reusing the layer input values as well as the weights. In the most extreme case, one would simply compute the entire result in parallel, which would mean that the weights could be completely static and every input would also only need to be stored (although dynamically updated per data set) exactly once. However, in such an architecture, most of the processing cycles would remain unused, which would result in an extremely inefficient resource usage, even though the data reuse is maximized. In consequence, it is necessary to find a way to divide the computations that reuses the data efficiently while also taking into account how many cycles are there for processing and that is feasible from an engineering point of view.

As example, computing an entire output channel at once would also allow an extremely efficient weight reuse, as all processing units could get the same set of weights for the current channel, and the weights would need to change only cycle by cycle. The input reuse would even still be maximized, as every output channel requires the complete input data. However, such a scheme would work well only if the number of processing cycles matches the number of output channels well, and otherwise could become extremely inefficient.

As other extreme, one could make use of nearly all of the available processing cycles if the computations are divided on a per-output-element basis, as for the fully-connected layer. While feasible in principle, this would make it very complicated to find and implement an efficient way of sharing inputs and weights between processing elements.

As compromise between the optimization of the data sharing and cycle usage, we decided to use one ’row’ of data as smallest unit for distribution to different processing units, which we accordingly name ’row units’ (RUs). We identify a row by its height and channel index, and it spans the entire width of the volume, see figure 3.

Generally, with $N_{\mathrm{OE}}$ output elements to be computed and $C$ cycles for processing, one requires at least N_{\mathrm{PU}}=\big{\lceil}\frac{N_{\mathrm{OE}}}{C}\big{\rceil} processing units. With a row-wise division, there are $N_{\mathrm{OE}}=H_{\mathrm{O}}\cdot D_{\mathrm{O}}$ output elements to be computed. This means that $N_{\mathrm{OE}}$ is easily in the order of one hundred, while $C$ is expected to be 20 or significantly smaller in case of the ATLAS detector trigger. If we use these values as assumption for ’large’ convolutional layers, the cycle efficiency (i.e. ratio of non-idling row unit cycles) is guaranteed to be at least 80% even in the worst case, and up to 100% in the best, which further improves with lower $C$ or larger layers. This is still not as good as what could be obtained for a per-element division, but it turns out that even for a row-wise division, one already needs to resort to a relatively complex design for combining the high cycle efficiency with an extensive data reuse, which is explained in the following.

The first level of data reuse is facilitated by the row unit itself: Neighboring output elements share a range of their inputs, and all of their weights. Accordingly, it is only necessary to provide one set of weights for the subunits within a row unit that compute the individual output elements, and only one range of the input data which is then shared internally.

A second level of data reuse can be introduced between row units: If row units are guaranteed to always process output rows belonging to the same channels in parallel, then it is also possible to let them share their weights throughout the entire computation.

Furthermore, it is also possible to use input sharing between the $N_{\mathrm{RU}}=big\lceil\frac{H_{\mathrm{O}}\cdot D_{\mathrm{O}}}{C}big\rceil$ row units. For that, we want to introduce the concept of a ’slice’, which contains all rows at a given height index, i.e. it is identified by the height index and spans the entire width and depth of the data volume, see figure 3. Row units that process rows from neighboring slices can share part of their inputs, as neighboring output slices require $H_{\mathrm{K}}-1$ common input slices for their results.

Until here, there is ’for-free’ data sharing within a row unit, and there are known conditions for inter-row-unit data sharing. Both belong to the spatial data reuse rather than the temporal data reuse. As final step, it is necessary to find a way to assign different output rows to row unit processing cycles, such that both the inter-row spatial conditions are met as well as possible and that temporal data reuse is also facilitated.

We developed the following scheme: Initially, all row units begin processing data for the same output channel, but of directly neighboring slices, i.e. row unit $i$ begins computing the result of the first channel of output slice $i$. Cycle by cycle, every row unit progresses to the next channel of the same slice. All row units begin computing outputs for the next free range of slices when all rows within the current slice range have been covered.

This scheme has the advantage that it maximizes both the input and the weight sharing between row units, and additionally, it is only necessary to load a new range of inputs when the row units move on to the next range of output slices, i.e. only once every $D_{\mathrm{O}}$ cycles. This means a drastic reduction of the amount of different inputs that need to be loadable, and therefore is very beneficial for the resource utilization, as this effects both registers for data storage and LUTs for input selection.

An example for this scheme is shown in figure 4(a). Generally, any row unit can process at most $C$ rows, which can be part of N_{\mathrm{RU,Sl}}=\big{\lfloor}\frac{C}{D_{\mathrm{O}}}\big{\rfloor} complete slices and $N_{\mathrm{RU,Rem}}=C\bmod D_{\mathrm{O}}$ remaining rows.

At this point, it is necessary to distinguish between a regular and an irregular case: In the regular case, there are at most as many slices as can be completely processed by the given number of row units, i.e. $N_{\mathrm{Sl}}\leq N_{\mathrm{RU}}\cdot N_{\mathrm{RU,Sl}}$. This means that it is possible to process all results for any slice by only a single RU. It might be necessary to have some RUs process one slice more than others, we attribute those which process an extra slice as ’long’, the others as ’short’.

In the ’irregular’ case, it is necessary to also use at least some of the $N_{\mathrm{RU,Rem}}$ remainder cycles of some row units, and the allocation scheme becomes much more complicated (see section A.1).

Code structure

The main part of the 2D convolutional layer are the row units. These are provided with all of the weights and inputs they need. For a row with $W_{\mathrm{O}}$ output positions, there are just as many subunits for computing the respective output values. Every subunit is structured as in figure 5: The inputs that are relevant for the given output position are weighted and accumulated per input channel from top to bottom channel, and partial results are pipelined individually for each kernel position. At the end, the total result for the given output position is produced by adding the partial results from all kernel positions. The top-to-bottom input scheme was used to adapt to the output scheme, which primarily also works from top to bottom channel, in order to typically allow faster data propagation between layers.

The rest of the layer was designed in an attempt to reduce the resource utilization (rather than frequency). A simplified structural example for the regular case is shown in figure 4(b). There is one input buffer memory, which has external write controlling and stores each of the (2D multi-channeled) inputs exactly once. The buffer memory is followed by working memories, which have control inputs for selecting when data is written to them and from which range of inputs data is taken. There always is a ’long’, and if ’short’ RUs are present, also a ’short’ working memory. The ’long’ working memory contains all input slices required for the ’long’ RUs. The ’short’ working memory contains all extra input slices required for the ’short’ RUs, i.e. those which are not required in any ’long’ RU. We chose to differentiate the working memories into these two subunits because the ’short’ memory needs to load one set of inputs less from the buffer memory, as the ’short’ RUs finish their computation one cycle earlier, and therefore this structure can reduce the resource utilization. In figure 4(b), each working memory output signal corresponds to an input slice, i.e. this spans the entire width and all channels, but has a fixed height index (which varies over time, while different ranges of inputs are loaded into the working memories).

To save further resources, we grouped RUs which always process the same output channel during any given cycle together, and gave all of them only a single weight memory. In the regular case, this means that there is only one set of weight memories for all ’long’ and one set for all ’short’ RUs. Details on the more complicated irregular case are described in appendix A.1.

As for the fully-connected layer, the row unit results are finally connected to all output positions where they produce results during any cycle (see allocation scheme), and the output row write enable signal is managed by a controller, which also controls the working memory input selection and write enabling and the weight memories.

3.4 2D max-pooling layers

For default-stride pooling layers, where there is no input sharing, it is neither necessary nor possible to resort to as elaborate schemes as for the convolutional layers. Since all inputs are required only once, there is no way of saving resources or input accesses, and there is no need to use complicated row allocation patterns. For simplicity reasons, the concept of output rows and row units was still maintained, but the row allocation was simply done from top to bottom channel, top to bottom slice in an interleaved manner (see figure 6(a)).

Contrary to the convolutional layers, it was already possible to implement different padding options for the pooling layer: These include padding ’valid’ and ’same’ (as in Keras), and a new mode ’unchanged’.101010’Valid’ padding dismisses ’incomplete’ pooling inputs at the high-index edges, ’same’ symmetrically extends the input space to only have complete pooling inputs, ’unchanged’ extends only on the high-index edges. Internally, padding is realized by the extension or truncation of the input signals. The latter will not increase the utilization significantly, as constant propagation will be detected and the corresponding logic parts removed/simplified by Vivado.

Code structure

As for the convolution layer, the basic units of this layer are the row units. A row unit gets all inputs required to compute a single output row. Since pooling usually does not extend over multiple channels, this means that a row unit only requires a range of input rows of a single channel for each cycle.

A structural schematic of the design is shown in figure 6(b). Similar to the convolutional layers, a buffer memory is used, from which the row unit working memories loads a new input slice during each cycle. This process is driven by a controller entity, which also controls the output write enabling. Row unit results are again multicast to any position in which they are needed at some cycle.

3.5 Activation functions

Apart from the obvious linear/identity activation, we currently support the rectified linear unit (relu) activation, which is quite commonly used in (convolutional) deep neural networks, and has the advantage that it can be implemented with very little cost on FPGAs.

By explicitly instantiating on a design primitive level, we were able to guarantee a relu implementation that requires either \big{\lfloor}\frac{B}{2}\big{\rfloor} LUTs or $B-1$ FFs per relu unit working on $B$ bit values. For the LUT-based implementation, we used that Xilinx UltraScale+ LUTs support dual-output use for up to five common inputs. The relu is then obtained by providing two input value bits and the input value sign bit to a LUT, which either replicates the two input bits on its outputs (positive sign) or sets the outputs to zero (negative sign). The FF-based implementation makes use of the FDRE primitive111111This realizes a D-type flip-flop with a synchronous reset and a clock enable input, the latter was tied to a logical ’1’. for every non-sign input value bit. The flip-flops simply store the input value bits if the sign is positive and reset to ’0’ if the sign is negative, i.e. the sign bit acts as reset control bit. This is an elegant demonstration of how advanced storage primitives can be used to implement not only storage, but also simple computations, which can save LUT resources for more complex operations.

Other activations can be implemented in the future. One way of doing this is by either value-based look-up tables or by value-derivative-based look-up tables. For example, with 16 bit activation unit input values, one could use the 8 most significant bits to look up an interpolation base value and a first derivative, and then add the first derivative multiplied by the eight least significant input bits to the base value. With the Xilinx US+ architecture, both look-ups could be done at an approximate LUT cost of $2\cdot 2^{8-6}\cdot 8=64$, the first two comes from two look-ups, the factor $2^{8-6}$ from the cost for looking up one bit for an $2^{8}$-deep address space and the eight comes from an assumed eight bits which are looked up for the base value and first derivative. The multiplication of two 8 bit values and final addition would cost approximately 70 LUTs, leaving a total of $\sim 140$ LUTs per activation unit to implement a relatively precise look-up with 256 sample points and linear interpolation in between.121212The exact LUT cost depends on some details, but less than 200 is seen as reasonable assumption for many cases. Characteristics like base value and derivative saturation could be further exploited for an even decreased LUT requirement. Figure 7 shows a case study, where we linearly interpolated the comparably complicated tanh and sigmoid functions. To demonstrate the loose precision requirements, we used only sixteen sample points within the shown intervals, and rounded to a granularity of only $2^{-6}$ during all steps of the interpolation (i.e. the value and derivate samples themselves were rounded, the interpolation position was rounded and multiplication and addition results were rounded). This case study nicely demonstrates that even with only very low precision and very few sample points, it is possible to have a surprisingly accurate approximation of non-linear activations. In these cases, it would be possible to implement the look-up of both base value and derivative with approximately ten LUTs and the multiplication and addition with coarsely 50 LUTs. Given that there would usually be only a single activation unit at the end of a many-DSP pipeline, one can expect an extra utilization of much less than 10 LUTs per DSP even for a precise interpolation.

4 Network creation toolkit

4.1 Layer synchronization considerations

When connecting fully-connected layers to fully-connected layers, it is only reasonable to select a pipeline parallelization factor $P$ in the successor layer that equals the number of neuron units $N_{\mathrm{NU}}$ in the predecessor layer. By that, every input is used exactly when it is made available, and no extra buffering structures or delays are necessary.

When connecting layers with ’arbitrary’ input and output schemes, there is no sense in starting the second layer earlier or later than necessary to run without interruption. This means that it is necessary to compute the minimum start delay that allows uninterrupted computation of the successor layer. Minimum start delay determination can easily be done if the output and input pattern of the involved layers are known, i.e. at which cycle which results/inputs are produced/needed. By element-wise subtraction of the ’needed’ pattern from the ’available’ pattern, one can take the largest positive value as minimum necessary delay for continuous computation.

However, with a given start delay, it might happen for some layer sequences that an input is already overwritten before it is needed for the last time.131313For example the 2D convolution might have multiple load operations from its buffer memory and can therefore create such a situation. These cases can be solved by the introduction of individual extra delays for all affected input values. The extra delay for each input value can be computed by adding the start delay to the input ’needed’ scheme, adding $C-1$ to the ’available’ scheme, subtracting the latter from the former and then introducing the respective extra delay for all values which yield a positive result. This does not increase the layer-to-layer latency.

Further delays that need to be taken into account are possible extra delays introduced by the application of activation functions and delay offsets between the layer enable signal and when that layer actually expects the first input. The latter do not influence the network latency itself, but need to be considered for asserting the layer enable signals at the right moments, which is automatically done in our toolkit.

In the special case of flattening, it is currently necessary to ’regularize’ the fully-connected layer input, because the fully-connected layer has no buffer memory, and therefore it is necessary to ensure that the inputs are updated in the exact order in which the computations propagate through the pipeline. This requires an extra auxiliary layer resulting in one extra cycle delay. The same effect could in the future be obtained by re-ordering the inputs/weights of the fully-connected layer, such that the inputs which are available first after flattening are also required first. This was not yet implemented because the regularization approach is more general, and it was understood only lately that a mere re-ordering would be sufficient, if the fully-connected layer parallelization $P$ is set to the number of simultaneously produced result values from the last 2D layer.

4.2 Network creation

The Python-based toolkit for automated network creation and the VHDL library files can be obtained via email from the authors and is distributed as open source software. The starting point for the network implementation on FPGA is a trained Keras network. Supported network architectures consist of the previously described layers and activation functions. An arbitrary sequence of 2D multi-channeled layers can be followed by flattening and an arbitrary sequence of fully-connected layers, or a fully-connected-only network can also be used.

Before the network can be implemented for the FGPA, it is necessary to specify/customize various design parameters. These include the precision (i.e. integer and fractional bits) of the input value and weight representation, which can even be customized on a layer-wise basis. Apart from specifying layer input and output bit widths, it is even possible to configure how intermediate values are treated, which occur for partial results being passed between DSPs and from the last DSP within a pipeline to a potential logic-based adder. Other parameters include pipelining and routing behavior, for example whether the relu activation is implemented in LUTs or FFs, if data flowing between DSPs is routed through the dedicated neighbor connections or the general fabric routing, or where registers are placed in arithmetic and logic pipelines.

Based on this information and the trained Keras network, the toolkit can be used for creating the VHDL network top file, auxiliary package file for configuration constants, network simulation testbench file and network data files for initializing control and weight memories from a Keras network. The VHDL files can finally be included in a project together with the network library files and are ready for use. Then, it is only necessary to load the respective weights and controller data into the memories during execution, after which inference is possible.

5 Results

Given our target design parameters, i.e. $40\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$ data frequency, at most $600\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$ to $900\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$ processing frequency and the Xilinx XCVU9P FPGA, we considered a possible network size of at most multiple $10^{4}$ MACs, to constrain ourselves to reasonable layer and network sizes for further studies. Typical deep neural networks in use by ATLAS start at a few $10^{3}$ MACs for fully-connected networks aiming at tagging $W$-Bosons and Top-Quarks [11], but can easily reach the order of several $10^{7}$ MACs for e.g. quark/gluon identification using convolutional networks on calorimeter images [12], as they have not been optimized for low resource usage.

For any design, timing closure was characterized by the ’worst negative slack’ (WNS), which can be thought of as difference between the signal propagation time estimated by Vivado and the target clock period. Timing characteristics apart from the WNS were met for all implemented designs. All designs were implemented with the ’out of context synthesis’. The bit lengths of the layer input values and results were set to 14 (6.8), and where occurring weights were set to 10 (2.8) bits length.141414If any, we observed less than 1% accuracy degradation in all of our MNIST digit recognition test networks with that precision choice. Fewer or more bits might be necessary for satisfying accuracy with other networks, with an approximately linear effect on the utilization when both bit widths are scaled simultaneously.

5.1 Isolated layers

All implementations were done with a target clock period of $1.563\text{\,}\mathrm{n}\mathrm{s}$, reflecting the $f_{\mathrm{P}}=$640\text{,}\mathrm{M}\mathrm{H}\mathrm{z}$$ processing frequency that are ideally achievable even with speed grade 1 US+ DSPs, which corresponds to $C=16$ for an assumed data rate of $40\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$, as in the ATLAS experiment. Therefrom, we derived the minimum processing clock period $T_{\mathrm{P,min}}=T_{\mathrm{P}}-T_{\mathrm{WNS}}$ of a design. Designs which failed with a large WNS might show better results for larger $T_{\mathrm{P}}$, because for better-suited $T_{\mathrm{P}}$, Vivado can typically reach better $T_{\mathrm{P,min}}$.

Fully-connected layers

We implemented fully-connected layers for $N_{\mathrm{I}},N_{\mathrm{N}}\in\{8,16,24,\allowbreak 32,\allowbreak 50,\allowbreak 64,\allowbreak 75,\allowbreak 100,\allowbreak 128\}$, $C\in\{10,16\}$. As expected, the DSP utilization represents what was designed for, i.e. $N_{\mathrm{I}}\cdot N_{\mathrm{NU}}$. BRAM utilization is at most 0.5\cdot\big{\lceil}\frac{N_{\mathrm{I}}}{P}\big{\rceil}\cdot N_{\mathrm{NU}}\cdot\big{\lceil}\frac{P}{3}\big{\rceil}+0.5, since for $P>3$ and 10 bit weights, multiple BRAM (sub-)units are necessary for most stage weight memories. For larger layers, this corresponds to approximately one BRAM per 5 DSPs, which is tolerable given the resource ratios in the US+ device family. For all of these designs, at most 4 LUTs and 23 FFs were required per DSP, with less than 10 FFs per DSP in almost 75% of the designs. Larger designs tended to have lower per-DSP LUT and FF utilization, due to positive scaling effects. Compared to $\sim 170$ LUTs and $\sim 340$ FFs per DSP in the XCVU9P FPGA, both values are negligible. Layer size in terms of MACs against maximum frequency is shown in figure 8. Depending on the device choice, it is possible to have up to multiple thousand MACs in a layer, before the implemented design rather than the device specification becomes limiting.

2D convolutional layers

The exact structure and therefore resource utilization of 2D convolutional layers depends strongly on the architectural parameters. To give a good overview, we implemented for some hand-picked parameters, to ensure all architectural features are covered at least once, and did a grid implementation for $H_{\mathrm{I}}\times W_{\mathrm{I}}\in\{(10,10),(15,15),(16,16),(20,20),(25,25)\}$, $I_{\mathrm{D}}\in\{1,2,4,6\}$, $H_{\mathrm{K}}\times W_{\mathrm{K}}\times N_{\mathrm{K}}\in\{(2,2),(3,3),(4,4)\}\times\{{2,4,6}\}$, $C\in\{10,16\}$ with a veto for these if more than 3000 DSPs (i.e. almost half of the XCVU9P DSPs) are needed for the given single layer. By design, no BRAM is used. The number of DSPs needed is $N_{\mathrm{RU}}\cdot W_{\mathrm{O}}\cdot H_{\mathrm{K}}\cdot W_{\mathrm{K}}\cdot D_{\mathrm{I}}$. For the ratios of LUTs and FFs to DSPs, we achieved 6 to 61 (with 75% of the designs below 25, and 95% below 40) and 10 to 77 (with 75% of the designs below 40 and 90% below 50), respectively, which is still a good result given the available resource ratios. Again, larger layers tend to require fewer LUTs and FFs per DSP. A plot of layer size in terms of MACs against maximum frequency is shown in figure 8. It turned out that for comparable size, convolutional layers typically reach lower maximum frequencies than fully-connected layers. This is a result of the optimization for resource utilization rather than timing at a time when the resource consumption was not yet known. Critical signal paths could already be identified, and we expect to be able to make it possible to trade a higher frequency for more utilization in the future.

2D max-pooling layers

Test implementations of layers included $H_{\mathrm{P}}\times W_{\mathrm{P}}\in\{(2,2),(3,3),(4,4)\}$, $H_{\mathrm{I}}\times W_{\mathrm{I}}\in\{(10,10),(16,16),(20,20),(30,30),(40,40)\}$, $D_{\mathrm{I}}\in\{1,2,4\}$, $C\in\{10,16\}$, padding ’same’. Per input value, between 5 and 16 LUTs were required, with 70% of the values below 8, and 18 to 29 FFs, with 65% of the values below 20. Hence, pooling layer utilization is negligible for computationally reasonable input sizes, with even only up to 52k LUTs and 130k FFs needed for as many as 6400 inputs.151515That many inputs would already be difficult to handle with convolutional layers, depending on the exact layer parameters. Layer size in terms of inputs against maximum frequency is shown in figure 8. Small layers met the target frequency of $640\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$ without any problems, some larger designs fell below that line, but not significantly. Future alternative designs could yield increased frequencies, but this was not yet studied, as an improved implementation of 2D convolutions appears more urgent.

5.2 Example networks

We tested four basic network architectures, where in the following we use I, C, P, F and D for 2D input, 2D convolutional, 2D pooling, flattening and fully-connected layers respectively. Architecture ArcA has the layer sequence I-C-P-F-D-D, ArcB has I-C-P-C-F-D-D, ArcC has I-C-P-C-F-D-D-D and ArcD has I-C-C-C-F-D-D-D. For each architecture, we trained various networks with varying layer parameters for the MNIST digit recognition task. The last fully-connected layer always had 10 neurons to classify the ten digits. A range of values for $C$ was also scanned for each network, where as data frequency $f_{\mathrm{D}}$ we chose $40\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$, and as (target) processing frequency $C\cdot f_{\mathrm{D}}$. We used a precision choice of 6.8 and 2.8 for values and weights, for which at most a sub-percent order of classification accuracy loss was observed, compared to CPU inference using float32 as data type. Some of the implementation results are shown in table 5.2.