Bit Transition Reduction by Data Transmission Ordering in NoC-based DNN Accelerator

TL;DR

This paper introduces a data transmission ordering method for NoC-based DNN accelerators that significantly reduces bit transitions, thereby lowering link power consumption across various data types and network configurations.

Contribution

It proposes a novel '1'-bit count-based ordering method with mathematical proof, and two data ordering strategies, affiliated and separated, to effectively reduce bit transitions in DNN workloads.

Findings

Up to 20.38% BT reduction for floating-point-32 data without NoC.

Up to 55.71% BT reduction for fixed-point-8 data without NoC.

Up to 40.85% BT reduction in NoC-based DNN accelerators.

Abstract

As Deep Neural Networks (DNN) are becoming essential, Network-on-Chip (NoC)-based DNN accelerators gained increasing popularity. To save link power in NoC, many researchers focus on reducing the Bit Transition (BT). We propose '1'-bit count-based ordering method to reduce BT for DNN workloads. We provide a mathematical proof of the efficacy of proposed ordering. We evaluate our method through experiments without NoC and with NoC. Without NoC, our proposed ordering method achieves up to 20.38% BT reduction for floating-point-32 data and 55.71% for fixed-point-8 data, respectively. We propose two data ordering methods, affiliated-ordering and separated-ordering to process weight and input jointly or individually and apply them to run full DNNs in NoC-based DNN accelerator. We evaluate our approaches under various configurations, including different DNN models such as LeNet and DarkNet, various NoC sizes with different numbers of memory controllers, random weights and trained weights, and different data precision. Our approach efficiently reduces the link power by achieving up to 32.01% BT reduction for floating-point-32 data and 40.85% BT reduction for fixed-point-8 data.

Tables

Table 1. TABLE I : BT reduction without NoC

Weights	Flit size (bit)	BTs per flit		BT Reduction rate
		Baseline	Ordered
Float-32 random	32$\times$8	113.27	90.18	20.38%
Fixed-8 random	8$\times$8	31.01	22.42	27.70%
Float-32 trained	32$\times$8	112.80	91.46	18.92%
Fixed-8 trained	8$\times$8	30.55	13.73	55.71%

Equations

$$P(t_{OneLink})$$

$$=1-\frac{(32-x)(32-y)}{32\times 32}-\frac{xy}{32\times 32}$$

$$E=32\times P(t)=x+y-\frac{xy}{16},$$

$$E_t=\sum _{i=1}^N{E}_i=\sum _{i=1}^N{x}_i+\sum _{i=1}^N{y}_i-\frac{{\sum }_{i=1}^N(x_i\times y_i)}{16}$$

$$F=\sum (x_i\times y_i).$$

Full text

I Introduction

Deep Neural Networks (DNN) have been widely deployed for extensive services. To efficiently run DNN models, DNN accelerators are essential. In recent years, Network-on-Chip (NoC)-based architectures [1, 2] have gained significant popularity in designing DNN accelerators.

The power of NoC is a critical constraint on performance as NoCs can consume a significant portion (10-30%) of total chip power [3]. Inside the NoC, power is consumed by buffers, arbiters, switches, and links. Links constitute a significant portion of the overall NoC’s power [4]. Links consume nearly 17% of the overall NoC power [5], and Banerjee et al. [6] report that link power accounts for around 42% of the interconnect power.

Many techniques are proposed to mitigate link power consumption. Paul et al. [7] introduce a voltage control approach to lower link power. Reza [3] dynamically adjusts link bandwidths based on system utilization and application workload characteristics. Chen et al. [8] reduces the active link number.

Reducing bit transitions (BTs) is a popular research area aimed at reducing link power, focusing more on the data transmitted over links instead of link design itself. BT is defined as a change from ’0’ to ’1’ or ’1’ to ’0’. Pagliari et al. [9] extend the differential encoding with bounded approximations to achieve BT reduction. SILENT [10] is proposed to reduce BT on serial links. Sarman et al. [11] employ the Delta encoding technique to transform original data into a representation with fewer ${}^{\prime}1^{\prime}$ bits, thereby reducing BTs. As demonstrated by these works, the research community widely recognizes BT reduction as a key optimization target.

Mittal and Nag [12] note that most existing BT reduction techniques have been proposed for CPUs. Given the vastly different architectures and design philosophies between CPUs and AI accelerators, Mittal and Nag [12] emphasize the need for developing accelerator-specific techniques. Targeting BT reduction in NoC-based DNN accelerators (NOC-DNA), we conclude our contribution as follows:

Mathematical analysis We provide a mathematical proof of the efficacy of our ’1’-bit count-based descending ordering approach in reducing BTs. 2. 2.

Effective ordering for DNN workloads: Our approach achieves up to 40.85% BT reduction when running full DNN workload in NoC, and we utilize the inherent order-insensitivity in DNN convolutional and linear layers. 3. 3.

Validation across various configurations: We comprehensively validate our method through: without NoC and with NoC, different NoC sizes, different DNN models, and different data formats.

The rest of the paper is organized as follows: Sec. II covers the related work. Sec. III provides problem definitions and a solution with mathematical proof. Sec. V presents the experimental results and Sec. VI concludes the paper.

II Related work

The most straightforward way to reduce link power is to adjust the link design. Mineo et al. [13] introduce that the voltage swing in links significantly impacts power consumption. Paul et al. [7] control the voltage to reduce link power. Configurable architecture is also used to reduce link power. Reza [3] utilizes reinforcement learning to dynamically decide the link bandwidth. Chen et al. [8] reduce the active link number by deactivating associated L2 cache banks.

Reducing BT absorbs researchers’ interest [9], and BT reduction rates vary for different designs and applications. Silent [10] proposed a serialized low-energy transmission coding for NoC and reduced switching activities by 32.35%. Stan et al. [14] reduce the average number BT by 25% compared to unencoded data. But it needs extra lines on bus. Pagliari * et al.* [9] achieve bit reduction rate from 42.73% for transmitting document files (.doc). Ghosh * et al.* [15] proposed a method utilizing the correlations in the data to reduce self-transitions and achieve an average reduction of 25%. These two methods require prior information about the whole dataset.

Firstly, our method is not a bus-encoding method and operates without additional links. It also requires no prior information about the whole dataset. Secondly, while most existing coding approaches are designed for CPUs [12], our ordering methods are designed for DNN workload.

III BT reduction problem and mathematical solution

III-A Problem definition

In NoCs, BTs occur when consecutive flits with differing bit patterns (’0’→’1’ or ’1’→’0’) traverse the links. The problem is defined as **minimizing the expected number of BTs between two consecutive flits **.

We first consider comparing two 32-bit numbers passing the same 32-bit links. Assuming the first number contains $x$-bits of ’1’, and the second number contains $y$-bits of ’1’, where $x$ and $y$ are non-negative integers between 0 and 32, Eq. (1) shows the probability of bit transition on each single-bit link.

[TABLE]

Assuming different bits are independent and identically distributed, the expectation of the BT is Eq. (2). We present expectations of BT across different x and y in Fig. 1.

[TABLE]

Then we expand from comparing two 32-bit numbers to comparing two flits, containing a series of numbers. Assuming each flit contains $N$ 32-bit numbers, $x_{i}$ and $y_{i}$ representing the count of ${}^{\prime}1^{\prime}$ bit in $i$th number in flit $f_{1}$ and flit $f_{2}$, respectively ($1\leq i\leq N$). The total expectation of BTs is

[TABLE]

Since the data is determined by the application and is therefore fixed, the multisets of $x$ and $y$ are also fixed. Hence, regardless of changing positions of 32-bit numbers, $\sum_{i=1}^{N}x_{i}+\sum_{i=1}^{N}y_{i}$ is constant. The goal of minimizing $E_{t}$ is to find an arrangement solution for $x$ and $y$ to maximize

[TABLE]

.

III-B * ’1’-bit count-based optimal strategy*

III-B1 Count-based strategy

The intuitive way is to go through all options to find the best $F$ in Eq. (4). However, for inserting 2N values in two N-value flits, there are $(2N)!$ options. For 128-bit link with fixed-8 data, we have overall $32!$ options, which is larger than $2.6\times 10^{35}$ options. Instead of exhaustive search, we propose a count-based ordering to maximize $F$ in Eq. (4) as described below.

1. (1)

Given two N-number flits, $f_{1}$ and $f_{2}$, the values of the numbers are $[V_{1_{oldf1}};V_{2_{oldf1}};...V_{N_{oldf1}}]$ in $f_{1}$ and $[V_{1_{oldf2}};V_{2_{oldf2}};...V_{N_{oldf2}}]$ in $f_{2}$. The ’1’-bit counts of each number in $f_{1}$ and $f_{2}$ represented as $[x_{1_{old}};x_{2_{old}};...x_{N_{old}}]$ and $[y_{1_{old}};y_{2_{old}};...y_{N_{old}}]$, respectively. The initial order of both the $x$ and $y$ series is arbitrary—for instance, $[x_{2_{old}}>x_{3_{old}}>y_{1_{old}}...]$, with no inherent ordering. 2. (2)

After applying count-based ordering, we rearrange the values to be $[V_{1_{f1}};V_{2_{f1}};...V_{N_{f1}}]$ and $[V_{1_{f2}};V_{2_{f2}};...V_{N_{f2}}]$ to ensure a strictly decreasing sequence of ’1’bit counts: $[x_{1}>y_{1}>x_{2}>y_{2}...>x_{N}>y_{N}]$.

Our transformation only changes the order of each number $V$ and maintains strict numerical equivalence between the original and reordered sets. For example, we are just changing the numbers of previous flits from $[V_{A};V_{B}]$ in $f1$ and $[V_{C};V_{D}]$ in $f2$ to be $[V_{A};V_{C}]$ in $f1$ and $[V_{B};V_{D}]$ in $f2$.

III-B2 Proof with inductive steps

We provide mathematical proofs with inductive steps to show that this ordering is a globally optimal ordering. Given two flits carrying N-element sequences and the series of’1’-bit counts are $X$ and $Y$, the initial state is that the values in $X$ and $Y$ are randomly placed:

1. (1)

Local Pairwise Optimization: For any two number $[V_{i_{oldf1}};V_{j_{oldf1}}]$ in first flit and two numbers $[V_{i_{oldf2}};V_{j_{oldf2}}]$ in flit 2, where $i<j$, let their initial bit-counts be $[x_{i_{old}},x_{j_{old}}]$ in flit $f1$ and $[y_{i_{old}},y_{j_{old}}]$ in flit $f2$. Enforcing the ordering constraint $[x_{i}>y_{i}>x_{j}>y_{j}]$ maximizes the pairwise product sum: $(x_{i}*y_{i}+x_{j}*y_{j})>=(x_{i_{old}}*y_{i_{old}}+x_{j_{old}}*y_{j_{old}})$. This inequality holds trivially and can be easily verified through exhaustive enumeration, as there are only four elements. Hence, to maximize $\sum xy$ in Eq. (4) for these four numbers, we should order the values to be $V_{i_{f1}};V_{j_{f1}}$ and$V_{i_{f2}};V_{j_{f2}}$ with corresponding ’1’-bit count being $[x_{i};x_{j}]$ and $[y_{i};y_{j}]$, where $[x_{1}>y_{1}>x_{2}>y_{2}]$. 2. (2)

Global Optimization: Iteratively applying this rule to all pairs $(i,j)$ until convergence yields a globally optimal ordering: $[x_{1}>y_{1}>x_{2}>y_{2}...>x_{N}>y_{N}]$.

As proof, we demonstrate the ordering’s optimality; the choice of sorting algorithms (Bubble Sort/Bitonic Sort/Merge Sort) to achieve the ordering is not discussed. This proof provides clear and essential guidance in two-flit scenarios; we evaluate our methods later with the complex interleaving of flits in NoC carrying real DNN workloads.

IV Data transmission ordering for NOC-DNA

A typical neuron calculation in NOC-DNA involves the inputs and weights. By dividing the inputs into the left half and the weights into the right half of the flit as shown in the right side of Fig. 2, we can focus solely on the bit comparison between the weights part or on the whole flits.

We present the basic idea of arranging data to reduce BTs in Fig. 3(a). The weights are ordered following their own descending order of ’1’-bit count, highlighted by the arrows. According to different ways to process inputs and weights, we propose two methods:

• •

Affiliated-ordering: weights, together with inputs, are ordered based on the ’1’-bit counts of weights.

• •

Separated-ordering: weights and inputs are ordered according to their own ’1’-bit counts, respectively.

IV-A Affiliated-ordering

Affiliated-ordering arranges the weights in descending order based on their ’1’-bit count. Simultaneously, the inputs do not follow their own ordering trend, and no arrows for inputs. Instead, inputs remain affiliated with corresponding weights.

In Fig. 4, we do not order the padded zeros to highlight the ordering of inputs and weights. Before affiliated-ordering, original inputs and weights are randomly placed. After ordering based on ’1’-bit counts, we change the positions of each weight, and the indices are out of order. Crucially, the inputs are changed, but the indices of inputs and weights in corresponding positions are kept paired. The disadvantage is that it only gains BT reduction from the weights’ part, and the advantage is that it maintains the paired relationship between weights and inputs.

IV-B Separated-ordering

In separated-ordering, as illustrated in Fig. 3(b), the key difference is that the inputs are independently placed according to their own order based on ’1’-bit count (highlighted by arrows), instead of pairing with the weights.

In Fig. 4, the indices of inputs and weights in corresponding positions are not the same, meaning that the inputs and weights are not paired. This separated-ordering method reduces not only the weight part’s BT but also the input part’s BT.

IV-C Advantages related to DNN and NoC

IV-C1 Low overhead for recovering data

Conventional data encoding schemes require both encoding and decoding steps. Our data transmission ordering method not only reduces BT but also keeps the values’ integrity, which can allow no decoding process in NOC-DNA.

For affiliated-ordering, only the ordering operation is performed. No subsequent deordering is required because the convolutional and linear layers in DNNs exhibit order invariance: the output remains unchanged when reordering paired weights and inputs. One example is shown in Fig. 5. As long as the input and weight are paired, the output result will not be changed and is equal to the sum of the products of corresponding inputs and weights, $A\times a+B\times b+\cdots+I\times i$. Since affiliated-ordering preserves the pairing relationship between inputs and weights, we don’t need to recover the data.

For separated-ordering, just a minimal-bit-width index is required, and this costs less than complex bit-wise operations for traditional decoding methods.

IV-C2 Near off-chip memory placement

We place the ordering unit near off-chip memory instead of inside the NoC to sort the data. The off-chip resource constraints are less stringent, and Li et al. [16] have proposed modern technology for in-memory sorting. Additionally, our ordering unit’s count is the same as the number of memory controllers (MCs) and much smaller than the number of routers.

IV-C3 Hide extra latency

In DNN accelerators, the outputs computed at each layer are not immediately consumed by the next layer, as there are still many tasks of the current layer pending. This creates a sufficient time window for ordering data before they serve as the next layer’s inputs. The layer-level interval effectively hides ordering latency from the critical path.

V Experiment results

First, we evaluate the ordering technique without NoC to show the effectiveness in simple scenarios. Then we evaluate our ordering method on NocDAS [2], a NOC-DNA, where flits carry actual DNN data and undergo complex and real router-to-router transmission as shown in Fig. 7.

Our designed BT recording method is shown in Fig. 8. The current flit and the previous are compared, and BTs are accumulated across the entire NoC. BT recording is solely for performance evaluation, and the flit storage and BT summation should not be considered overheads.

V-A BT reductions without NoC

We generate 10,000 packets without NoC and measure the BTs of random comparisons between flits. Real weights are used as the payloads of flits, including trained LeNet weights and randomly initialized weights. Zeros are padded when the weight’s kernel size doesn’t exactly match the flit size. Two common data types, 32-bit floating-point (float-32) and 8-bit fixed-point (fixed-8), are used.

We present one example in Fig. 9. The y-axis is the flit ID and eight squares in each row represent eight weights in one flit, while the number inside the square indicates the ’1’-bit count of each weight. The data are descending ordered by their ’1’-bit count, as shown in the right of Fig. 9.

V-A1 Summary of BT reduction without NoC

We summarize the average BT reductions in Tab. I. Proposed data transmission ordering method achieves a BT reduction of 20.38% for float-32 random weights and 18.92% for float-32 trained weights. For fixed-8 weights, the impact of ordering is more significant and the reductions are 27.70% and 55.71%.

V-A2 Bit distribution and transition probability

The value distribution of weights has been widely studied; however, the bit-level distribution of values remains an interesting topic.

For float-32 weights, we present the bit distribution across the 32 positions at the top of Fig. 10. We can obviously distinguish the sign bit, 8-bit exponent, and 23-bit mantissa as they have different patterns. The first sign bit is around 0.5. For the exponent part, we observe that the beginning bits (2-6 in the x-axis) and the last three bits (7-9 in the x-axis) show notable differences. For the mantissa part (beyond position 9 on the x-axis), the results reveal that random data exhibits more uniform bit distribution, whereas trained data displays some fluctuations in tail bits.

As for BT probability at the bottom in Fig. 10, the orange color is lower than the blue color, showing that the ordering method makes the transition probability lower than without ordering.

For BT of fixed-8 data, as shown in Fig. 11, the impact of ordering is more significant. In the bottom of Fig. 11, the gap between the orange and blue bars is significant, particularly for trained fixed-point 8-bit data. The distinct gap in the bottom-right of Fig. 11 align with the highest BT reduction rate 55.71% in Tab. I for fixed-8 data.

V-B BT reduction with real traffic in NoC

We conducted experiments to run a complete DNN on a NOC-DNA [2], where numerous real packets are injected and transferred with interleaving of flits from different packets. The NoC use X-Y routing, 4 virtual channels (VCs) with a 4-flit-depth buffer per VC. Dual data formats and two link bandwidths are used: 512-bit link for 16 float-32 values, and 128-bit link for 16 fixed-8 values.

• •

We evaluate three configurations: baseline without ordering (O0), affiliated-ordering (O1), and separated-ordering (O2) as introduced in Sec. IV-B.

• •

We explore different NoC sizes, including a default 4$\times$4 NoC with 2 memory controllers (MCs) (MC2), 8$\times$8 NoC with 4 MCs (MC4), and 8$\times$8 NoC with 8 MCs (MC8).

• •

We run experiments on different DNN models, including LeNet and a DarkNet-like model. We reduce the input size for DarkNet to $64\times 64\times 3$ to speed up the simulation.

V-B1 Different NoC sizes

For the absolute BT number in Fig. 12, the green MC4 NoC results in more BTs compared to other NoC sizes. The reason is that more routers per MC increase the hops that a flit must traverse, and more hops lead to more overall BTs.

When comparing BT reduction rates, affiliated-ordering effectively reduces BT from 12.09% to 18.58% for floating-32 data and from 7.88% to 17.75% for fixed-8 data. Separated-ordering achieves BT reduction rate from 23.30% to 32.01% for float-32 data and from 16.95% to 35.93% for fixed-8 data. A consistent trend emerges: affiliated-ordering achieves significant BT reductions, while separated-ordering consistently delivers the highest reductions.

V-B2 Different DNN models

We achieve up to 35.93% BT reduction for LeNet, and up to 40.85% for DarkNet in Fig. 13. Aligning with the results across varying NoC sizes, the separated-ordering method achieves the highest BT reduction.

V-C Hardware design for ordering unit

To evaluate the overhead of our method, we present the hardware design of affiliated-ordering unit. This unit can be used for separated-ordering with double time consumption. As shown in Fig. 14, classic Single-instruction-multiple-data Within A Register (SWAR) to count ’1’-bits (Pop-count) and bubble sort are implemented. We synthesize a NoC generated from Constellation [17] using Synopsys DC and compare routers with our units in Tab. II. Our ordering unit occupies only 12.91 kGE (thousand gate equivalents). While a single router consumes 16.92 mW, one ordering unit requires much less power (2.213 mW). The deordering overhead is very light and the count of ordering units equals the number of MCs - for instance, four units in an 8×8 NoC containing 64 routers.

We synthesize physical links with Innovus and obtain the bit transition energy of $0.173pJ$. Banerjee et al. [6] presents bit transition energy of $0.532pJ$. Assuming half of the 128-bit links transit for an 8x8 NoC with 112 inter-router links, the overall link power under 125MHz is $0.173pJ/bit*\frac{128bits}{2}*112*125MHz=155.008mW$ for our design and 476.672 mW using Banerjee’s [6] link model. The link power and energy depend on many factors, such as physical link length and chip size, material capacitance, etc. We provide it merely to give an intuitive impression. With the BT reduction of 40.85% by our method, link power is reduced from 155.008 mW to 91.688 mW or from 476.672 mW to 281.951 mW.

VI Conclusion

We propose a data transmission ordering approach to reduce BT in NOC-DNA. We present a mathematical analysis showing that a descending ordering based on ’1’-bit count is globally optimal for minimizing the BT. Then we evaluate our method with and without NoC. Two ordering methods, affiliated-ordering and separated-ordering, are proposed. Across diverse configurations—including varying NoC sizes, DNN models, and data formats, our approach achieves significant BT reductions. We also present the hardware implementation to assess hardware overhead.

The affiliated-ordering method achieves a BT reduction rate of up to 18.58% for float-32 data and 17.75% for fixed-8 data. Separated-ordering effectively reduces BT, achieving up to 32.01% for float-32 data and 40.85% reduction in BTs for fixed-8 data. A single ordering unit only occupies 12.91 kGE of area. Four ordering units consume 8.852 mW total power, while 64 routers consume 1083.18 mW. Combining and comparing this work with other BT reduction works, running more DNN workloads can be explored in the future.