Distributed Deep Convolutional Neural Networks for the Internet-of-Things

TL;DR

This paper presents a methodology for deploying distributed deep convolutional neural networks on IoT devices, optimizing for low latency and resource constraints to enable real-time, autonomous decision-making.

Contribution

It introduces a formal optimization-based design methodology for allocating CNN execution across IoT units considering memory and processing constraints.

Findings

Optimized CNN distribution reduces decision latency.

Supports multiple data sources and CNNs simultaneously.

Enables real-time, autonomous IoT applications.

Abstract

Severe constraints on memory and computation characterizing the Internet-of-Things (IoT) units may prevent the execution of Deep Learning (DL)-based solutions, which typically demand large memory and high processing load. In order to support a real-time execution of the considered DL model at the IoT unit level, DL solutions must be designed having in mind constraints on memory and processing capability exposed by the chosen IoT technology. In this paper, we introduce a design methodology aiming at allocating the execution of Convolutional Neural Networks (CNNs) on a distributed IoT application. Such a methodology is formalized as an optimization problem where the latency between the data-gathering phase and the subsequent decision-making one is minimized, within the given constraints on memory and processing load at the units level. The methodology supports multiple sources of data as well as multiple CNNs in execution on the same IoT system allowing the design of CNN-based applications demanding autonomy, low decision-latency, and high Quality-of-Service.

Tables

Table 1. TABLE I: The memory demand m j subscript 𝑚 𝑗 m_{j} , the computational load c j subscript 𝑐 𝑗 c_{j} , and the memory K j subscript 𝐾 𝑗 K_{j} required to store the intermediate representations of the ResNet 101  [ 4 ] , with a 4B data type. When there are two values for m j subscript 𝑚 𝑗 m_{j} and c j subscript 𝑐 𝑗 c_{j} in repeated sequence of layers, the former one refers to the first repetition, the latter to all the subsequent repetitions.

	Layer ($j$)	$m_j$ (KB)	$c_j$ (${10}^6$ mult.)	$K_j$ (KB)
s	Source (Im. 224x224x3)	-	-	602.12
$1_1$	7x7 conv, 64, /2	37.63	118.01	3 211.26
$1_2$	3x3 max pool, /2	-	1.81	802.82
	3x $\{\begin{array}{cc}\text{1x1 conv, 64}& \\ \text{3x3 conv, 64}& \\ \text{1x1 conv, 256}& \end{array}$	16.38–65.54	12.85–51.38	802.82
2		147.46	115.61	802.82
		65.54	51.38	3 211.26
		786.43	616.56	3 211.26
	4x $\{\begin{array}{cc}\text{1x1 conv, 128}& \\ \text{3x3 conv, 128}& \\ \text{1x1 conv, 512}& \end{array}$	131.07–262.14	25.69–51.38	401.41
3		65.54	115.61	401.41
		262.14	51.38	1 605.63
		2228.2	757.86	1 605.63
	5x $\{\begin{array}{cc}\text{1x1 conv, 256}& \\ \text{3x3 conv, 256}& \\ \text{1x1 conv, 1024}& \end{array}$	524.29–1 048.58	25.69–51.38	200.71
4		2 359.30	115.61	200.71
		1 048.58	51.38	800.82
		21 757.95	950.53	800.82
	6x $\{\begin{array}{cc}\text{1x1 conv, 256}& \\ \text{3x3 conv, 256}& \\ \text{1x1 conv, 1024}& \end{array}$	1 048.58	51.38	200.71
5,6,7		2 359.30	115.61	200.71
		1 048.58	51.38	800.82
		26 738.69	1 156.06	800.82
	3x $\{\begin{array}{cc}\text{1x1 conv, 512}& \\ \text{3x3 conv, 512}& \\ \text{1x1 conv, 2048}& \end{array}$	2 097.15–4 194.30	25.69–51.38	100.35
8		9 437.18	115.61	200.71
		4 194.30	51.38	800.82
		51 380.22	629.41	800.82
$9_1$	7x7 avg pool	-	0.10	8.19
$9_2$	fc 1000	8 388.61	2.08	4

Table 2. TABLE II: The memory demand m j subscript 𝑚 𝑗 m_{j} , the computational load c j subscript 𝑐 𝑗 c_{j} , and the memory K j subscript 𝐾 𝑗 K_{j} required to store the intermediate representations of four (EX–)CNNs, with a 4B data type and the Early-Exit layer marked with an asterisk. In that layer, K j subscript 𝐾 𝑗 K_{j} represents the dimensions of the representation sent to the layer j + 1 𝑗 1 j+1 when the classification is not taken at layer j 𝑗 j .

	Layer ($j$)	$m_j$ (KB)	$c_j$ (${10}^6$ mult.)	$K_j$ (KB)
s	Source (Im. 28x28x3)	-	-	9.41
$1_1$	5x5 conv, 64	19.20	3.76	200.70
$1_2$	2x2 pool, /2	-	0.05	50.18
2*	gc1 (fc 384,192,10)	19 570.18	4.89	50.18
$3_1$	5x5 conv, 64	409.60	20.07	50.18
$3_2$	2x2 pool, /2	-	0.01	12.54
4	fc 384	4 816.90	1.20	1.54
5	fc 192	294.91	0.07	0.77
6	fc 10	7.68	$2\cdot {10}^{-3}$	0.04

Table 3. (a)

	Layer ($j$)	$m_j$ (KB)	$c_j$ (${10}^6$ mult.)	$K_j$ (KB)
s	Source (Im. 28x28x3)	-	-	9.41
$1_1$	5x5 conv, 64	19.20	3.76	200.70
$1_2$	2x2 pool, /2	-	0.05	50.18
2*	gc1 (fc 384,192,10)	19 570.18	4.89	50.18
$3_1$	5x5 conv, 64	409.60	20.07	50.18
$3_2$	2x2 pool, /2	-	0.01	12.54
4	fc 384	4 816.90	1.20	1.54
5	fc 192	294.91	0.07	0.77
6	fc 10	7.68	$2\cdot {10}^{-3}$	0.04

Table 4. (b)

	Layer ($j$)	$m_j$ (KB)	$c_j$ (${10}^6$ mult.)	$K_j$ (KB)
s	Source (Im. 227x227x3)	-	-	618.35
$1_1$	11x11 conv, 96, /4	139.78	105.42	1161.60
$1_2$	3x3 pool, /2	-	0.31	279.94
$2_1$	5x5 conv, 256	1 229.82	223.95	746.50
$2_2$	3x3 pool, /2	-	0.39	173.06
3*	gc1(fc 128,64,2)	22185.22	5.55	173.06
4	3x3 conv, 384	3 540.48	149.52	259.58
5	3x3 conv, 384	2 655.74	112.14	259.58
$6_1$	3x3 conv, 256	1 770.50	74.76	173.06
$6_2$	3x3 pool, /2	-	0.08	36.86
7	fc 4096	151 011.39	37.75	16.38
8	fc 4096, 2	67 158.02	16.78	0.01

Table 5. TABLE III: The maximum memory usage m ¯ i subscript ¯ 𝑚 𝑖 \bar{m}_{i} (defined as half of the available RAM), and the million ( 10 6 superscript 10 6 10^{6} ) multiplications per second e i subscript 𝑒 𝑖 e_{i} s (defined as a tenth of the clock cycles performed in one second) of a few off-the-shelf IoT units.

	Node ($i$)	${\overline{m}}_i$ (KB)	$e_i$ (${10}^6$ mult.)
S1	STM32H7	512	40
B1	BeagleBone AI	524 288	360
O1	OrangePi Zero	131 072	480
R1	Raspberry Pi 3B+	524 288	560

Table 6. TABLE IV: Single-(EX–)CNN configuration results with N = 30 𝑁 30 N=30 STM32H7 and Raspberry Pi 3B+ units in the 50%-50% scenario. The figure of merit (mean ± plus-or-minus \pm std) is the latency t 𝑡 t , i.e., the transmission time t t subscript 𝑡 𝑡 t_{t} plus the processing time t p subscript 𝑡 𝑝 t_{p} .

		Latency $t$ (ms)						Latency $t$ (ms)
	L	$t_t$	$t_p$	$t=t_t+t_p$			L	$t_t$	$t_p$	$t=t_t+t_p$
5-layer CNN	1	$7.49\pm 0.36$	$44.93\pm 0.00$	$52.42\pm 0.36$		AlexNet	1	$203.06\pm 36.85$	$1257.71\pm 0.00$	$1460.77\pm 36.85$
	2	$1.78\pm 0.21$	$44.93\pm 0.00$	$46.71\pm 0.21$			2	$127.81\pm 28.49$	$1257.71\pm 0.00$	$1385.52\pm 28.49$
	3	$0.48\pm 0.14$	$44.93\pm 0.00$	$45.41\pm 0.14$			3	$98.75\pm 26.63$	$1257.71\pm 0.00$	$1356.46\pm 26.63$
	4	$0.37\pm 0.11$	$44.93\pm 0.00$	$45.30\pm 0.11$			4	$95.82\pm 26.54$	$1257.71\pm 0.00$	$1353.53\pm 26.54$
	C	$0.28\pm 0.11$	$44.93\pm 0.00$	$45.21\pm 0.11$			C	$72.49\pm 24.08$	$1257.71\pm 0.00$	$1330.20\pm 24.08$
6-layer EX-CNN	1	$5.87\pm 0.27$	$7.27\pm 0.00$	$13.14\pm 0.27$		EX-AlexNet	1	$129.42\pm 28.29$	$598.92\pm 81.81$	$728.35\pm 86.17$
	2	$0.34\pm 0.10$	$7.27\pm 0.00$	$7.61\pm 0.10$			2	$93.70\pm 24.49$	$596.19\pm 0.00$	$689.89\pm 24.49$
	3	$0.29\pm 0.10$	$7.27\pm 0.00$	$7.57\pm 0.10$			3	$72.10\pm 22.08$	$596.19\pm 0.00$	$668.29\pm 22.08$
	4	$0.28\pm 0.10$	$7.27\pm 0.00$	$7.55\pm 0.10$			4	$72.07\pm 22.08$	$596.19\pm 0.00$	$668.26\pm 22.08$
	C	$0.28\pm 0.10$	$7.27\pm 0.00$	$7.55\pm 0.10$			C	$71.84\pm 22.07$	$596.19\pm 0.00$	$668.03\pm 22.07$

Table 7. (a)

		Latency $t$ (ms)						Latency $t$ (ms)
	L	$t_t$	$t_p$	$t=t_t+t_p$			L	$t_t$	$t_p$	$t=t_t+t_p$
5-layer CNN	1	$7.49\pm 0.36$	$44.93\pm 0.00$	$52.42\pm 0.36$		AlexNet	1	$203.06\pm 36.85$	$1257.71\pm 0.00$	$1460.77\pm 36.85$
	2	$1.78\pm 0.21$	$44.93\pm 0.00$	$46.71\pm 0.21$			2	$127.81\pm 28.49$	$1257.71\pm 0.00$	$1385.52\pm 28.49$
	3	$0.48\pm 0.14$	$44.93\pm 0.00$	$45.41\pm 0.14$			3	$98.75\pm 26.63$	$1257.71\pm 0.00$	$1356.46\pm 26.63$
	4	$0.37\pm 0.11$	$44.93\pm 0.00$	$45.30\pm 0.11$			4	$95.82\pm 26.54$	$1257.71\pm 0.00$	$1353.53\pm 26.54$
	C	$0.28\pm 0.11$	$44.93\pm 0.00$	$45.21\pm 0.11$			C	$72.49\pm 24.08$	$1257.71\pm 0.00$	$1330.20\pm 24.08$
6-layer EX-CNN	1	$5.87\pm 0.27$	$7.27\pm 0.00$	$13.14\pm 0.27$		EX-AlexNet	1	$129.42\pm 28.29$	$598.92\pm 81.81$	$728.35\pm 86.17$
	2	$0.34\pm 0.10$	$7.27\pm 0.00$	$7.61\pm 0.10$			2	$93.70\pm 24.49$	$596.19\pm 0.00$	$689.89\pm 24.49$
	3	$0.29\pm 0.10$	$7.27\pm 0.00$	$7.57\pm 0.10$			3	$72.10\pm 22.08$	$596.19\pm 0.00$	$668.29\pm 22.08$
	4	$0.28\pm 0.10$	$7.27\pm 0.00$	$7.55\pm 0.10$			4	$72.07\pm 22.08$	$596.19\pm 0.00$	$668.26\pm 22.08$
	C	$0.28\pm 0.10$	$7.27\pm 0.00$	$7.55\pm 0.10$			C	$71.84\pm 22.07$	$596.19\pm 0.00$	$668.03\pm 22.07$

Table 8. (b)

		Latency $t$ (ms)						Latency $t$ (ms)
	L	$t_t$	$t_p$	$t=t_t+t_p$			L	$t_t$	$t_p$	$t=t_t+t_p$
5-layer CNN	1	$74.82\pm 3.94$	$45.16\pm 0.57$	$119.98\pm 4.06$		AlexNet	1	$2042.48\pm 401.64$	$1265.89\pm 141.47$	$3308.37\pm 432.91$
	2	$17.75\pm 2.08$	$44.93\pm 0.00$	$62.68\pm 2.08$			2	$1298.23\pm 332.66$	$1263.16\pm 115.58$	$2561.39\pm 361.81$
	3	$4.49\pm 0.89$	$45.08\pm 0.46$	$49.57\pm 1.03$			3	$1009.79\pm 334.87$	$1260.44\pm 81.77$	$2270.22\pm 351.10$
	4	$3.64\pm 0.87$	$44.93\pm 0.00$	$48.57\pm 0.87$			4	$976.71\pm 317.48$	$1263.16\pm 115.58$	$2239.87\pm 348.56$
	C	$2.80\pm 0.88$	$44.93\pm 0.00$	$47.73\pm 0.88$			C	$750.02\pm 322.44$	$1260.44\pm 81.77$	$2010.46\pm 339.52$
6-layer EX-CNN	1	$58.72\pm 2.47$	$7.27\pm 0.01$	$65.99\pm 2.47$		EX-AlexNet	1	$1304.74\pm 310.87$	$598.93\pm 81.91$	$1903.66\pm 324.56$
	2	$3.53\pm 1.15$	$7.27\pm 0.00$	$10.80\pm 1.15$			2	$947.04\pm 258.12$	$596.19\pm 0.00$	$1543.23\pm 258.12$
	3	$3.04\pm 1.10$	$7.27\pm 0.00$	$10.31\pm 1.10$			3	$728.19\pm 221.12$	$596.19\pm 0.00$	$1324.38\pm 221.12$
	4	$2.90\pm 1.10$	$7.27\pm 0.00$	$10.17\pm 1.10$			4	$727.87\pm 221.10$	$596.19\pm 0.00$	$1324.06\pm 221.10$
	C	$2.88\pm 1.10$	$7.27\pm 0.00$	$10.16\pm 1.10$			C	$725.53\pm 220.95$	$596.19\pm 0.00$	$1321.72\pm 220.95$

Table 9. TABLE V: The multi CNN configuration results with N = 30 𝑁 30 N=30 STM32H7 and Raspberry Pi 3B+ units and two 5-layer CNNs, in the three scenarios, with the Wi-Fi 4 and Wi-Fi Halow transmission technologies. The figure of merit (mean ± plus-or-minus \pm std) is the latency t 𝑡 t , i.e., the transmission time t t subscript 𝑡 𝑡 t_{t} plus the processing time t p subscript 𝑡 𝑝 t_{p} and is summed over all the CNNs.

			10 – 90 Latency $t$ (ms)				50 – 50 Latency $t$ (ms)				90 – 10 Latency $t$ (ms)
		L	$t_t$	$t_p$	$t=t_t+t_p$		$t_t$	$t_p$	$t=t_t+t_p$		$t_t$	$t_p$	$t=t_t+t_p$
	No shared layers	1	$14.7\pm 0.3$	$89.9\pm 0.0$	$104.6\pm 0.3$		$15.6\pm 1.1$	$89.9\pm 0.1$	$105.4\pm 1.1$		$19.4\pm 5.5$	$634.4\pm 417.0$	$653.8\pm 413.6$
		2	$3.4\pm 0.1$	$89.9\pm 0.0$	$93.3\pm 0.1$		$3.8\pm 0.6$	$89.9\pm 0.0$	$93.6\pm 0.6$		$10.2\pm 5.0$	$317.1\pm 401.5$	$327.3\pm 399.7$
		3	$0.9\pm 0.1$	$89.9\pm 0.0$	$90.7\pm 0.1$		$1.1\pm 0.4$	$89.9\pm 0.0$	$90.9\pm 0.4$		$4.0\pm 2.9$	$198.0\pm 240.0$	$201.9\pm 242.4$
		4	$0.7\pm 0.1$	$89.9\pm 0.0$	$90.6\pm 0.1$		$0.8\pm 0.3$	$89.9\pm 0.0$	$90.7\pm 0.3$		$4.0\pm 4.5$	$119.8\pm 67.0$	$123.7\pm 71.2$
		5	$0.5\pm 0.1$	$89.9\pm 0.0$	$90.4\pm 0.1$		$0.7\pm 0.3$	$89.9\pm 0.0$	$90.5\pm 0.3$		$3.0\pm 2.8$	$105.1\pm 34.1$	$108.1\pm 36.6$
	First two layers shared	1	$14.6\pm 0.1$	$89.9\pm 0.0$	$104.5\pm 0.1$		$15.4\pm 1.2$	$89.9\pm 0.1$	$105.3\pm 1.2$		$23.4\pm 9.0$	$452.8\pm 470.8$	$476.2\pm 467.4$
		2	$3.4\pm 0.1$	$89.9\pm 0.0$	$93.3\pm 0.1$		$3.7\pm 0.5$	$89.9\pm 0.0$	$93.5\pm 0.5$		$9.6\pm 8.6$	$250.6\pm 389.8$	$260.2\pm 388.0$
		3	$2.0\pm 0.0$	$89.9\pm 0.0$	$91.9\pm 0.0$		$2.2\pm 0.4$	$89.9\pm 0.0$	$92.1\pm 0.4$		$7.1\pm 5.2$	$115.9\pm 63.1$	$122.9\pm 65.8$
		4	$0.8\pm 0.0$	$89.9\pm 0.0$	$90.7\pm 0.0$		$0.9\pm 0.3$	$89.9\pm 0.0$	$90.8\pm 0.3$		$2.3\pm 1.6$	$90.5\pm 1.3$	$92.8\pm 2.2$
		5	$0.8\pm 0.0$	$89.9\pm 0.0$	$90.6\pm 0.0$		$0.8\pm 0.2$	$89.9\pm 0.0$	$90.7\pm 0.2$		$2.0\pm 1.4$	$90.2\pm 0.6$	$92.2\pm 1.7$

Table 10. (a)

			10 – 90 Latency $t$ (ms)				50 – 50 Latency $t$ (ms)				90 – 10 Latency $t$ (ms)
		L	$t_t$	$t_p$	$t=t_t+t_p$		$t_t$	$t_p$	$t=t_t+t_p$		$t_t$	$t_p$	$t=t_t+t_p$
	No shared layers	1	$14.7\pm 0.3$	$89.9\pm 0.0$	$104.6\pm 0.3$		$15.6\pm 1.1$	$89.9\pm 0.1$	$105.4\pm 1.1$		$19.4\pm 5.5$	$634.4\pm 417.0$	$653.8\pm 413.6$
		2	$3.4\pm 0.1$	$89.9\pm 0.0$	$93.3\pm 0.1$		$3.8\pm 0.6$	$89.9\pm 0.0$	$93.6\pm 0.6$		$10.2\pm 5.0$	$317.1\pm 401.5$	$327.3\pm 399.7$
		3	$0.9\pm 0.1$	$89.9\pm 0.0$	$90.7\pm 0.1$		$1.1\pm 0.4$	$89.9\pm 0.0$	$90.9\pm 0.4$		$4.0\pm 2.9$	$198.0\pm 240.0$	$201.9\pm 242.4$
		4	$0.7\pm 0.1$	$89.9\pm 0.0$	$90.6\pm 0.1$		$0.8\pm 0.3$	$89.9\pm 0.0$	$90.7\pm 0.3$		$4.0\pm 4.5$	$119.8\pm 67.0$	$123.7\pm 71.2$
		5	$0.5\pm 0.1$	$89.9\pm 0.0$	$90.4\pm 0.1$		$0.7\pm 0.3$	$89.9\pm 0.0$	$90.5\pm 0.3$		$3.0\pm 2.8$	$105.1\pm 34.1$	$108.1\pm 36.6$
	First two layers shared	1	$14.6\pm 0.1$	$89.9\pm 0.0$	$104.5\pm 0.1$		$15.4\pm 1.2$	$89.9\pm 0.1$	$105.3\pm 1.2$		$23.4\pm 9.0$	$452.8\pm 470.8$	$476.2\pm 467.4$
		2	$3.4\pm 0.1$	$89.9\pm 0.0$	$93.3\pm 0.1$		$3.7\pm 0.5$	$89.9\pm 0.0$	$93.5\pm 0.5$		$9.6\pm 8.6$	$250.6\pm 389.8$	$260.2\pm 388.0$
		3	$2.0\pm 0.0$	$89.9\pm 0.0$	$91.9\pm 0.0$		$2.2\pm 0.4$	$89.9\pm 0.0$	$92.1\pm 0.4$		$7.1\pm 5.2$	$115.9\pm 63.1$	$122.9\pm 65.8$
		4	$0.8\pm 0.0$	$89.9\pm 0.0$	$90.7\pm 0.0$		$0.9\pm 0.3$	$89.9\pm 0.0$	$90.8\pm 0.3$		$2.3\pm 1.6$	$90.5\pm 1.3$	$92.8\pm 2.2$
		5	$0.8\pm 0.0$	$89.9\pm 0.0$	$90.6\pm 0.0$		$0.8\pm 0.2$	$89.9\pm 0.0$	$90.7\pm 0.2$		$2.0\pm 1.4$	$90.2\pm 0.6$	$92.2\pm 1.7$

Table 11. (b)

			10 – 90 Latency $t$ (ms)				50 – 50 Latency $t$ (ms)				90 – 10 Latency $t$ (ms)
		L	$t_t$	$t_p$	$t=t_t+t_p$		$t_t$	$t_p$	$t=t_t+t_p$		$t_t$	$t_p$	$t=t_t+t_p$
	No shared layers	1	$147.2\pm 2.3$	$89.9\pm 0.3$	$237.2\pm 2.3$		$154.4\pm 11.5$	$91.2\pm 6.1$	$245.6\pm 14.1$		$187.0\pm 41.4$	$624.6\pm 416.9$	$811.6\pm 394.1$
		2	$34.5\pm 1.5$	$89.9\pm 0.0$	$124.4\pm 1.5$		$37.4\pm 5.3$	$90.0\pm 0.6$	$127.4\pm 5.4$		$95.3\pm 44.4$	$301.7\pm 383.1$	$397.0\pm 372.3$
		3	$8.8\pm 1.2$	$89.9\pm 0.3$	$98.8\pm 1.3$		$9.9\pm 2.7$	$90.4\pm 0.9$	$100.3\pm 3.0$		$32.9\pm 28.8$	$188.1\pm 228.8$	$221.0\pm 254.2$
		4	$7.0\pm 1.0$	$89.9\pm 0.0$	$96.9\pm 1.0$		$8.1\pm 2.6$	$89.9\pm 0.0$	$98.0\pm 2.6$		$37.5\pm 42.6$	$116.6\pm 64.0$	$154.1\pm 105.1$
		5	$5.3\pm 0.9$	$89.9\pm 0.0$	$95.2\pm 0.9$		$6.4\pm 2.7$	$89.9\pm 0.0$	$96.3\pm 2.7$		$28.6\pm 27.0$	$103.5\pm 32.6$	$132.1\pm 57.2$
	First two layers shared	1	$146.6\pm 1.6$	$89.9\pm 0.3$	$236.5\pm 1.7$		$152.3\pm 9.8$	$90.7\pm 1.2$	$243.1\pm 10.1$		$223.7\pm 63.6$	$454.4\pm 459.1$	$678.0\pm 434.8$
		2	$34.1\pm 0.7$	$89.9\pm 0.2$	$124.0\pm 0.8$		$36.0\pm 4.3$	$90.2\pm 0.9$	$126.2\pm 4.5$		$85.4\pm 72.2$	$252.5\pm 389.8$	$338.0\pm 381.7$
		3	$20.4\pm 0.7$	$89.9\pm 0.2$	$110.3\pm 0.7$		$21.9\pm 3.9$	$90.1\pm 0.6$	$112.1\pm 4.0$		$65.3\pm 46.3$	$117.0\pm 62.8$	$182.3\pm 97.5$
		4	$8.4\pm 0.3$	$89.9\pm 0.2$	$98.3\pm 0.4$		$9.0\pm 2.2$	$90.2\pm 0.9$	$99.2\pm 2.4$		$17.9\pm 14.1$	$92.1\pm 1.5$	$110.0\pm 14.4$
		5	$7.6\pm 0.3$	$89.9\pm 0.2$	$97.5\pm 0.3$		$8.2\pm 2.2$	$90.0\pm 0.5$	$98.2\pm 2.2$		$17.1\pm 14.1$	$91.0\pm 0.7$	$108.0\pm 14.2$

Table 12. TABLE VI: The single CNN configuration results with N = 50 𝑁 50 N=50 OrangePi Zero, BeagleBone AI, and Raspberry Pi 3B+ units (with probability 45%-45%-10%) and 1 ResNet CNN. The figures of merit (mean ± plus-or-minus \pm std) are the latency t 𝑡 t , i.e., the transmission time t t subscript 𝑡 𝑡 t_{t} plus the processing time t p subscript 𝑡 𝑝 t_{p} , and the number η 𝜂 \eta of IoT units used.

		Latency $t$ (ms)				Node usage $\eta$
	L	$t_t$	$t_p$	$t=t_t+t_p$		${\eta }_R$	${\eta }_O$	${\eta }_B$
	1	$361.85\pm 60.77$	$12059.62\pm 430.32$	$12421.47\pm 406.82$		$4.92\pm 1.84$	$4.05\pm 1.84$	$0.03\pm 0.17$
	2	$249.47\pm 69.22$	$11723.26\pm 319.91$	$11972.72\pm 313.71$		$3.59\pm 0.90$	$1.49\pm 0.95$	$0.01\pm 0.10$
	3	$183.34\pm 87.14$	$11645.13\pm 259.90$	$11828.47\pm 267.93$		$2.80\pm 0.55$	$0.55\pm 0.86$	$0.00\pm 0.06$
	4	$138.99\pm 85.02$	$11620.58\pm 236.18$	$11759.57\pm 252.58$		$2.14\pm 0.50$	$0.88\pm 0.55$	$0.00\pm 0.07$
	C	$127.85\pm 94.81$	$11602.91\pm 215.07$	$11730.76\pm 237.14$		$0.99\pm 0.11$	$0.08\pm 0.34$	$0.00\pm 0.00$

Table 13. TABLE VII: The multi CNN configuration results with N = 50 𝑁 50 N=50 OrangePi Zero, BeagleBone AI, and Raspberry Pi 3B+ units (with probability 45%-45%-10%) and 1 to 4 AlexNet (A) CNNs. The figures of merit (mean ± plus-or-minus \pm std) are the latency t 𝑡 t , i.e., the transmission time t t subscript 𝑡 𝑡 t_{t} plus the processing time t p subscript 𝑡 𝑝 t_{p} , and the number η 𝜂 \eta of IoT units used.

			Latency $t$ (ms)				Node usage $\eta$
		L	$t_t$	$t_p$	$t=t_t+t_p$		${\eta }_R$	${\eta }_O$	${\eta }_B$
	1 AlexNet	1	$206.22\pm 26.04$	$\mathrm{1\hspace{0.17em}390.95}\pm 54.58$	$\mathrm{1\hspace{0.17em}597.17}\pm 54.33$		$3.97\pm 1.51$	$4.01\pm 1.48$	$0.02\pm 0.15$
		2	$151.25\pm 30.93$	$\mathrm{1\hspace{0.17em}344.36}\pm 56.70$	$\mathrm{1\hspace{0.17em}495.61}\pm 60.89$		$3.00\pm 0.87$	$1.43\pm 1.08$	$0.01\pm 0.12$
		3	$125.55\pm 33.97$	$\mathrm{1\hspace{0.17em}328.63}\pm 51.73$	$\mathrm{1\hspace{0.17em}454.19}\pm 62.16$		$2.47\pm 0.63$	$0.64\pm 0.74$	$0.01\pm 0.12$
		4	$119.76\pm 34.24$	$\mathrm{1\hspace{0.17em}323.49}\pm 47.45$	$\mathrm{1\hspace{0.17em}443.25}\pm 59.89$		$1.96\pm 0.50$	$0.51\pm 0.73$	$0.01\pm 0.09$
		C	$97.88\pm 39.57$	$\mathrm{1\hspace{0.17em}311.71}\pm 44.03$	$\mathrm{1\hspace{0.17em}409.58}\pm 66.63$		$1.00\pm 0.11$	$0.17\pm 0.45$	$0.01\pm 0.09$
	3 AlexNet	1	$578.58\pm 50.35$	$\mathrm{4\hspace{0.17em}390.39}\pm 104.53$	$\mathrm{4\hspace{0.17em}968.97}\pm 119.87$		$5.07\pm 2.01$	$17.89\pm 1.72$	$1.05\pm 1.24$
		2	$410.26\pm 43.87$	$\mathrm{4\hspace{0.17em}211.45}\pm 148.26$	$\mathrm{4\hspace{0.17em}621.71}\pm 142.12$		$5.05\pm 1.97$	$7.72\pm 1.95$	$0.11\pm 0.38$
		3	$350.06\pm 45.97$	$\mathrm{4\hspace{0.17em}108.70}\pm 154.98$	$\mathrm{4\hspace{0.17em}458.76}\pm 152.25$		$4.84\pm 1.71$	$4.65\pm 1.98$	$0.04\pm 0.26$
		4	$349.40\pm 54.44$	$\mathrm{4\hspace{0.17em}035.36}\pm 146.02$	$\mathrm{4\hspace{0.17em}384.76}\pm 147.68$		$4.46\pm 1.34$	$3.13\pm 1.86$	$0.04\pm 0.26$
		C	$306.24\pm 76.75$	$\mathrm{3\hspace{0.17em}947.47}\pm 106.77$	$\mathrm{4\hspace{0.17em}253.71}\pm 144.39$		$2.86\pm 0.46$	$0.90\pm 1.33$	$0.02\pm 0.21$
	4 AlexNet	1	$772.23\pm 54.88$	$\mathrm{5\hspace{0.17em}960.88}\pm 138.87$	$\mathrm{6\hspace{0.17em}733.11}\pm 156.94$		$5.22\pm 2.20$	$21.73\pm 2.80$	$5.04\pm 2.99$
		2	$521.34\pm 36.80$	$\mathrm{5\hspace{0.17em}718.77}\pm 181.19$	$\mathrm{6\hspace{0.17em}240.11}\pm 175.43$		$5.22\pm 2.20$	$11.62\pm 2.20$	$0.19\pm 0.55$
		3	$437.76\pm 50.07$	$\mathrm{5\hspace{0.17em}572.27}\pm 206.36$	$\mathrm{6\hspace{0.17em}010.03}\pm 196.11$		$5.17\pm 2.12$	$7.47\pm 2.34$	$0.07\pm 0.40$
		4	$441.12\pm 59.81$	$\mathrm{5\hspace{0.17em}458.45}\pm 213.15$	$\mathrm{5\hspace{0.17em}899.57}\pm 201.50$		$4.99\pm 1.85$	$5.25\pm 2.36$	$0.10\pm 0.38$
		C	$401.83\pm 77.75$	$\mathrm{5\hspace{0.17em}288.65}\pm 150.73$	$\mathrm{5\hspace{0.17em}690.48}\pm 183.33$		$3.64\pm 0.74$	$1.81\pm 1.93$	$0.04\pm 0.30$

Table 14. TABLE VIII: The multi CNN configuration results with N = 50 𝑁 50 N=50 OrangePi Zero, BeagleBone AI, and Raspberry Pi 3B+ units (with probability 45%-45%-10%) and 1 to 4 EX-AlexNet CNNs. The figures of merit (mean ± plus-or-minus \pm std) are the latency t 𝑡 t , i.e., the transmission time t t subscript 𝑡 𝑡 t_{t} plus the processing time t p subscript 𝑡 𝑝 t_{p} , and the number η 𝜂 \eta of IoT units used.

			Latency $t$ (ms)				Node usage $\eta$
		L	$t_t$	$t_p$	$t=t_t+t_p$		${\eta }_R$	${\eta }_O$	${\eta }_B$
	1 AlexNet	1	$117.54\pm 17.66$	$666.38\pm 33.56$	$783.92\pm 36.27$		$4.08\pm 1.46$	$3.79\pm 1.45$	$0.13\pm 0.34$
		2	$93.87\pm 24.95$	$643.84\pm 34.46$	$737.71\pm 40.10$		$3.09\pm 0.82$	$1.51\pm 1.00$	$0.07\pm 0.25$
		3	$77.03\pm 25.44$	$641.06\pm 34.80$	$718.09\pm 41.04$		$2.66\pm 0.58$	$0.44\pm 0.67$	$0.02\pm 0.13$
		4	$74.56\pm 24.62$	$640.61\pm 33.94$	$715.17\pm 40.58$		$1.98\pm 0.49$	$0.44\pm 0.71$	$0.00\pm 0.06$
		C	$68.15\pm 24.14$	$639.05\pm 34.41$	$707.20\pm 42.28$		$1.00\pm 0.06$	$0.24\pm 0.49$	$0.01\pm 0.08$
	3 AlexNet	1	$335.42\pm 21.31$	$\mathrm{2\hspace{0.17em}078.03}\pm 54.60$	$\mathrm{2\hspace{0.17em}413.45}\pm 59.52$		$4.95\pm 2.08$	$17.26\pm 1.72$	$1.79\pm 1.29$
		2	$268.92\pm 38.28$	$1989.59\pm 72.05$	$2258.52\pm 72.76$		$4.91\pm 2.03$	$8.36\pm 1.82$	$0.55\pm 0.78$
		3	$222.56\pm 39.05$	$1971.77\pm 75.12$	$2194.33\pm 75.51$		$4.79\pm 1.90$	$4.84\pm 2.22$	$0.17\pm 0.59$
		4	$222.98\pm 39.87$	$1946.47\pm 73.10$	$2169.45\pm 76.50$		$4.37\pm 1.43$	$3.37\pm 2.00$	$0.11\pm 0.36$
		C	$211.27\pm 42.69$	$1914.79\pm 64.03$	$2126.06\pm 75.80$		$2.87\pm 0.51$	$1.08\pm 1.36$	$0.03\pm 0.17$
	4 AlexNet	1	$452.33\pm 29.81$	$2803.39\pm 67.85$	$3255.73\pm 78.62$		$4.95\pm 2.17$	$21.52\pm 2.72$	$5.53\pm 2.56$
		2	$346.37\pm 43.73$	$2689.37\pm 94.61$	$3035.74\pm 93.17$		$4.95\pm 2.17$	$12.39\pm 1.79$	$1.19\pm 1.35$
		3	$284.80\pm 46.16$	$2666.04\pm 100.07$	$2950.84\pm 96.52$		$4.93\pm 2.13$	$8.07\pm 2.15$	$0.72\pm 1.05$
		4	$289.94\pm 44.66$	$2619.44\pm 103.84$	$2909.38\pm 102.62$		$4.79\pm 1.89$	$5.59\pm 2.24$	$0.25\pm 0.68$
		C	$281.67\pm 50.74$	$2558.70\pm 98.07$	$2840.36\pm 105.54$		$3.63\pm 0.95$	$1.86\pm 2.12$	$0.12\pm 0.51$

Table 15. TABLE IX: Experimental benchmark results with equally spaced nodes and a 5-layer CNN. The figure of merit is the latency t 𝑡 t (transmission t t subscript 𝑡 𝑡 t_{t} plus processing t p subscript 𝑡 𝑝 t_{p} ).

	L	Case	Wi-Fi 4 Latency (ms)
			$t_t$	$t_p$	$t=t_t+t_p$
	4	Model	0.37	44.98	45.35
		Experimental	0.42	68.47	68.89

Equations

$$g_{u,j}=\{\begin{array}{ll}{p}_{u,j}-p_{u,j+1}& \text{ if }j<M_u\\ 1-{\sum }_{v=1}^{M_u-1}{g}_{u,v}& \text{ if }j=M_u\end{array}.$$

$${\alpha }_{u,i,j}=\{\begin{array}{ll}1& \text{if IoT unit i executes layer j of CNN u}\\ 0& \text{otherwise}\end{array},$$

$$\begin{gathered} \sum _{u=1}^C\sum _{i=1}^N\sum _{k=1}^N\sum _{j=1}^{M-1}{\alpha }_{u,i,j}\cdot {\alpha }_{u,k,j+1}\cdot p_{u,j+1}\cdot \frac{K_{u,j}}{\rho }\cdot d_{i,k} \\ +\sum _{i=1}^N{t}_i^{(p)}+t_s+t_f, \end{gathered}$$

$$\forall i\in {\mathbb{N}}_N$$

$$\forall u\in {\mathbb{N}}_C,\forall j\in {\mathbb{N}}_M$$

$$t_s$$

$$t_i^{(p)}$$

$$t_f$$

$$\frac{K_{u,j}}{\rho }\cdot d_{i,k},$$

$$\forall i\in {\mathbb{N}}_N{\alpha }_{u_1,i,j_1}={\alpha }_{u_2,i,j_2},$$

$$\forall i\in {\mathbb{N}}_N$$

$${\alpha }_{i,j}=\{\begin{array}{ll}1& \text{ if IoT unit i executes CNN layer j}\\ 0& \text{ otherwise}\end{array},$$

$$\sum _{i=1}^N\sum _{k=i}^N\sum _{j=1}^{M-1}{\alpha }_{i,j}\cdot {\alpha }_{k,j+1}\cdot \frac{K_j}{\rho }\cdot d_{i,k}+\sum _{i=1}^N{t}_i^{(p)}+t_s+t_f.$$

$$\forall i\in {\mathbb{N}}_N$$

$$\forall j\in {\mathbb{N}}_M$$

$$t_s$$

$$t_i^{(p)}$$

$$t_f$$

$$\sum _{i=1}^N\sum _{k=i}^N\sum _{j=1}^{M-1}{\alpha }_{i,j}\cdot {\alpha }_{k,j+1}\cdot p_{j+1}\cdot \frac{K_j}{\rho }\cdot d_{i,k}+\sum _{i=1}^N{t}_i^{(p)}+t_s+t_f,$$

$$t_s$$

$$t_i^{(p)}$$

$$t_f$$

$$\sum _{u=1}^C\sum _{i=1}^N\sum _{k=i}^N\sum _{j=1}^{M-1}{\alpha }_{u,i,j}\cdot {\alpha }_{u,k,j+1}\cdot \frac{K_{u,j}}{\rho }\cdot d_{i,k}+\sum _{i=1}^N{t}_i^{(p)}+t_s+t_f,$$

$$t_s$$

$$t_i^{(p)}$$

$$t_f$$

$$\forall i\in {\mathbb{N}}_N$$

$$\hat{m}$$

$$\hat{c}$$

$$\forall i\in {\mathbb{N}}_N$$

Full text

IEEE Copyright Notice

Copyright $\copyright$ 2021 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

Distributed Deep Convolutional Neural Networks for the Internet-of-Things

Simone Disabato1

Manuel Roveri1

Cesare Alippi13

Published in IEEE Transactions on Computers.

The code of the paper is available at https://github.com/simdis/distributedCNNs.

Please cite as:

S. Disabato, M. Roveri and C. Alippi, "Distributed Deep Convolutional Neural Networks for the Internet-of-Things," in IEEE Transactions on Computers, vol. 70, no. 8, pp. 1239-1252, 1 Aug. 2021, doi: 10.1109/TC.2021.3062227.

BibTex

@article{disabato2021distributed, title={Distributed deep convolutional neural networks for the internet-of-things}, author={Disabato, Simone and Roveri, Manuel and Alippi, Cesare}, journal={IEEE Transactions on Computers}, year={2021}, volume={70}, number={8}, pages={1239-1252}, doi={10.1109/TC.2021.3062227}, publisher={IEEE} }

1 Introduction

Deep learning (DL) represents the state-of-the-art in many recognition/classification applications [1, 2, 3, 4] and is characterized by processing architectures organized in a pipeline of layers providing a hierarchical representation [5]. However, such solutions are also characterized by a high computational load and memory occupation [6, 7, 8] and, for this reason, their use is mostly restricted to high-performance computing platforms [9, 10, 11].

For this reason, Internet-of-Things (IoT) systems, whose computational units are mostly constrained by limited processing, memory and energy capacities, have been rarely considered a viable technological solution for DL. In fact, currently, IoT units are generally considered as simple data-collectors acquiring and transmitting data to the Cloud for DL processing [12], with very few solutions (mostly based on approximation techniques) proposing machine learning and (most rarely) deep learning for embedded systems [8, 13, 14]. Unfortunately, the request for a remote processing to make decisions limits the effectiveness of the system as the “data production to decision making”-latency might not satisfy real-time constraints. The system closed-loop stability may even be compromised when remote connectivity between IoT units and Cloud is unavailable or limited in bandwidth [15]. Hence, applications requesting a (quasi) real-time decision/actuation cannot take advantage of remote Cloud-based processing of DL solutions.

The problem of reducing the complexity of DL solutions to match the technological constraints of IoT systems is becoming more and more relevant from both the scientific and technological perspective and solutions present in the literature address such a problem at different levels (i.e., ad-hoc hardware, approximation techniques, off-loading of DL and distributed DL). Despite the growing research interest, the problem of assigning a distributed DL solution on a network of IoT embedded devices is still open in the literature.

In this research direction, this paper introduces a methodology that receives technological constraints associated with (possibly heterogeneous) IoT units and the DL trained architectures, and provides the optimal distributed assignment of the DL computation (layers) to the IoT units by minimizing the data gathering to decision latency. In particular, the methodology has been tailored to Convolutional Neural Networks (CNNs), representing the state-of-the-art in image and video processing. Interestingly, these CNNs can operate sequentially on the processing pipeline (all layers are executed up to the final classification) [16, 4, 17] or can decide the processing path at run-time (skipping the execution of some layers) according to the information content brought by the input [18, 19]. Both cases are considered in the proposed methodology that, in addition, can be used with multiple CNNs running on the same network of IoT embedded devices (possibly sharing processing layers).

With respect to the literature, the main novel aspects of the proposed methodology can be summarized as:

• •

the methodology is able to take into account communication and computation capabilities as well as memory constraints of the IoT units;

• •

the methodology can support CNNs whose processing pipeline depends on the specific input image;

• •

multiple CNNs (possibly sharing processing layers) can be executed on the same network of IoT units.

The proposed methodology has been validated both on simulations and a real-world technological scenario, while the related code is made available to the scientific community111The code is available at https://github.com/simdis/distributedCNNs.. The paper is organized as follows: the related literature is analysed in Section 2, the research problem is formulated in Section 3, whereas the proposed methodology is described in Section 4. Results are detailed in Sections 5 and 6, while conclusions are drawn in Section 7.

2 Related Literature

In the related literature, the problem of reducing the computational and memory demand of DL solutions to match the technological constraints of IoT systems can be addressed at different levels. At the hardware level, the best technological performances with minimum power consumption are achieved by ad-hoc hardware computing platforms for DL as specific processors [20], or configurable FPGA solutions [21, 22]. As also pointed out by [23], such solutions require high design-skills and cannot be easily re-used. Another approach, that has shown a significant drop in computation time, is the adoption of GPUs [16, 10], TPUs [24], or neural hardware. However, both custom hardware and GPU/TPU solutions cannot be easily considered in pervasive IoT systems.

The reduction of DL solutions complexity can be achieved by considering approximation techniques. Such an approach allows to reduce the memory and/or computational demand at the expense of a decrease in the accuracy. [25, 26, 27] proposed compression techniques on already trained CNNs, by compressing the weights, by pruning whole or part of layers, and by adopting Huffman coding. Another approach is the quantization of layers’ weights [28], up to binary weights, also during the training phase [29, 30, 31]. Similarly, [8] introduced task-dropping and precision-scaling mechanisms to design application-specific approximated CNNs able to be executed in off-the-shelf embedded systems, but not in distributed IoT systems. [32], instead, studied approximation to reduce inference on pooling and normalization layers. An approach to reduce the inference time is those of Adaptive Early Exit CNNs [18] or Gate-Classification CNNs [19], where the classification output can be provided also at intermediate layers, skipping the remaining computation.

A different point of view is provided by the off-loading techniques for distributed computing systems. Here, the goal is not to reduce DL solutions’ complexity but to move computationally-intensive processing to high-performing units of the distributed system. For instance, a framework to optimally offload code in a pervasive system comprising mobile units is proposed in [33], by either minimizing the total communication latency or the overall energy consumption. Differently, [34] proposed a high-level programming language to design applications meant for Fog-Computing Sensor Networks able to hide the heterogeneity of computing nodes and their position in space. [35] proposed a low-complexity scheduler that increases the throughput in IoT clusters by relying on tasks duplication and splitting by taking into account communication and computation capabilities, but not IoT units memory constraints. Very few works present in the literature encompass the code-offloading of machine learning-based applications in pervasive systems, e.g., [36] where the classification/pattern recognition tasks of a wearable device are partially offloaded to other computing units (e.g., mobile phones). Similarly to our vision, here the priority in the offloading is to move code at first to the closest mobile devices and, then, if needed, to the Cloud.

The problem of distributing DL solutions has been recently addressed in the field of edge and fog computing. [37] introduced a distributed framework for CNNs operating in edge computing platforms, with the possibility of distributing the CNNs computation, mostly restricted to the Cloud, also to edge or local devices. Unfortunately, the usage of the Cloud is predominant in this work. In [15, 37] emerged the need to completely re-design the DL solutions to take into account hardware and physical constraints at application design-time, to make IoT systems a viable technological solution for DL. Such an approach is considered in some recent works addressing the problem of distributing machine/deep learning solutions onto a network of IoT/Edge devices [38, 39, 40, 41, 14]. In particular, [14] organizes the computation tasks by partitioning the convolutional layers vertically, with the possibility of sharing the computed features among parallel tasks. [40], instead, proposed a dynamic scheduler to assign the layers to edge units and reduce the communication data by relying on autoencoders.

Unfortunately, almost all these solutions rely on approximations of the original ML or DL solution to match the hardware and physical constraints of the devices. In addition, the processing pipelines of the machine/deep learning solutions introduced in these works are fixed at design-time, hence not being able to modify their execution at run-time according to the information content brought by the input. Finally, the proposed solutions are not meant to be executed on low-power embedded systems (e.g., STM32 or Arduino).

3 Problem Formulation

The IoT system comprises a set of $C$ data-acquisition units $\{s_{1},\ldots,s_{C}\}$ mounting cameras and providing the images to be classified by the $C$ CNNs (each CNN processes data coming from one data-acquisition unit), a set $\mathbb{N}_{N}=\{1,\dots,N\}$ of $N$ possibly heterogeneous IoT units implementing code execution, and one target unit $f$ receiving all the $C$ CNNs’ outcomes to make a decision or activate a reaction. Without any loss of generality, the $C$ data-acquisition units are assumed to only acquire images and do not participate in the computation. The $i$-th IoT unit $i\in\mathbb{N}_{N}$ is characterized by its own constraints w.r.t. maximum memory capacity $\bar{m}_{i}$ and available computation $\bar{c_{i}}$.

The IoT system is modeled as a graph $G(V,E)$ of nodes $V=\{\mathbb{N}_{N}\cup\left\{s_{1},\dots,s_{C},f\right\}\}$ and arcs $E$. An arc $e_{i_{1},i_{2}}$ between unit $i_{1}$ and $i_{2}$ exists in $E$ if $i_{2}$ is within the range of the transmission technology the IoT unit $i_{1}$ is equipped with222All the units in $V$ are assumed to share the same transmission technology with a fixed transmission data-rate. If the IoT units $i_{1}$ and $i_{2}$ adopt two different transmission technologies such that $i_{2}$ is within the transmission range of $i_{1}$, but $i_{1}$ is not inside the one of $i_{2}$, then $d_{i_{1},i_{2}}=1\not=d_{i_{2},i_{1}}$ (loss of symmetry property).. Let $d_{i_{1},i_{2}}$, for each $i_{1},i_{2}\in V$, be the hop-distance between units $i_{1}$ and $i_{2}$ of $V$ defined as the number of hops (communication steps), data need to take to reach $i_{2}$ from $i_{1}$. In other terms, distance $d_{i_{1},i_{2}}$ is the shortest communication path between units $i_{1}$ and $i_{2}$ within the graph $G$. Following the definition of shortest path in a graph, if no path between unit $i_{1}$ and $i_{2}$ exists, then $d_{i_{1},i_{2}}=+\infty$. We also assume that no isolated node exists, i.e., $d_{i_{1},i_{2}}<\infty$, for each $i_{1},i_{2}\in V$.

Let $M_{u}$, for each $u\in\mathbb{N}_{C}=\{1,\ldots,C\}$, be the number of layers characterizing the $u$-th CNN. Each layer $j$ of CNN $u$, for each $j\in\{1,\ldots,M_{u}\}$ and $u\in\mathbb{N}_{C}$, is characterized by a given memory demand $m_{u,j}$ and computation $c_{u,j}$. More specifically, the memory complexity $m_{u,j}$ (in Bytes) is defined as the number of weights that layer $j$ of CNN $u$ has to store multiplied by the size of the data type used to represent those parameters (typically the floating-point type occupying 4 Bytes), while the computational load $c_{u,j}$ is measured as the number of multiplications to be executed by that layer [8]. Let $K_{u,j}$, for each $u\in\mathbb{N}_{C}$ and $j\in\{1,\ldots,M_{u}\}$, be the memory occupation of the intermediate representation transmitted from layer $j$ to the subsequent layer $j+1$ of CNN $u$, and $K_{u,s}$ and $K_{u,M_{u}}$ be the memory occupation of the input image of CNN $u$ transmitted from source $s_{u}$ to the unit executing the first layer of the $u$-th CNN and the final classification provided by layer $M_{u}$ sent to the target unit $f$, respectively. In particular, $K_{u,M_{u}}$ is either the classification label or the posterior probability of the classes resulting from the softmax layer.

When the processing path of a CNN depends on the information content, such as in Adaptive [18] or Gate-Classification CNNs [19], classification completes as soon as enough confidence is achieved (the execution of the remaining layers is aborted). To achieve this goal, Early-Exit CNNs (EX-CNNs) are endowed with intermediate exit points, creating multiple paths within the CNNs each of which is characterized by a probability of being traversed. More formally, given a $M_{u}$-layer EX-CNN, let $p_{u,j}\in[0,1]$ be the probability that the $j$-th layer of the $u$-th CNN processes the input image333The probabilities $p_{u,j}$s can be estimated during CNN learning. and let $g_{u,j}\in[0,1]$, for each $u\in\mathbb{N}_{C}$ and $j\in\{1,\ldots,M_{u}\}$, be the probability that the computation ends at layer $j$ of the $u$-th CNN is

[TABLE]

The addressed problem is the optimal placement of the $C$ CNNs layers on the $N$ IoT units to minimize the latency in transmitting decisions about the images gathered by the $C$ sources to the target unit $f$.

4 The Proposed Methodology

This section introduces the proposed methodology for the optimal placement of CNNs on the IoT system. Such a methodology has been reformulated as a decision-making latency minimization optimization problem aiming at optimally assigning the layers of the $C$ CNNs to the $N$ IoT units. This optimization problem relies on the C$$N$$M variables $\alpha_{u,i,j}$ defined as:

[TABLE]

for each $u\in\mathbb{N}_{C}$, for each $i\in\mathbb{N}_{N}$ and for each $j\in\mathbb{N}_{M}=\{1,\ldots,M\}$, being $M=max\{M_{1},\ldots,M_{u}\}$ the maximum number of layers among the considered $C$ CNNs (i.e., the maximum depth of all CNNs).

Without loss of generality, the distances $d_{i_{1},i_{2}}$, for each $i_{1},i_{2}\in V$, can be precomputed, allowing us to define an integer quadratic optimization problem on variables $\alpha_{u,i,j}$s. As detailed in Section 3, $\{s_{1},\ldots,s_{C}\}$ and $f$ do not participate in the optimization problem since their task is to acquire images and receive the final classification, respectively. This assumption can be easily removed by considering additional IoT computing units in the same positions of $s_{i}$s and $f$.

The objective function to be minimized models the latency in making a decision by the $C$ CNNs placed on the IoT units, defined as the time occurring between images acquisition by sensor unit $s_{u}$s (size $K_{u,s}$) and final classifications $K_{u,M_{u}}$s are transmitted to unit $f$:

[TABLE]

such that

[TABLE]

and where

[TABLE]

The objective function in Eq. (3) comprises four different components of the latency:

(i) The source time $t_{s}$, defined in Eq. (8), required to transmit the images from the sources $s_{u}$s to the IoT units executing the first layer of the CNNs. Although the first layer is always reached, i.e., $p_{u,1}=1$ for each $u\in\mathbb{N}_{C}$, the term $p_{u,1}$ has been added to Eq. (8) to provide homogeneity in the formalization.

(ii) The transmission time of intermediate representations among the IoT units processing the CNN layers. More precisely, the transmission time of the intermediate representation of the $j$-th layer of the $u$-th CNN from unit $i$ to $k$ is

[TABLE]

where $\rho$ is the data-rate of the considered transmission technology and $d_{i,k}$ is the hop-distance between unit $i$ and $k$ as defined in Section 3. In Eq. (3) the transmission time is weighted by the probability $p_{u,j+1}$ that layer $j+1$ is executed right after layer $j$.

(iii) The processing time $t_{i}^{(p)}$ of the CNN layers on the IoT units. Specifically, the processing time of layer $j$ of CNN $u$ on the $i$-th IoT unit is approximated as the ratio between the computational demand $c_{u,j}$ that layer requires and the number of multiplications $e_{i}$ the IoT unit $i$ is able to carry out in one second444The $e_{i}$s encompass the number of available cores, the type of pipeline such cores implement to approach one operation per clock cycle, the presence or not of a GPU allowing to parallelize CNN operations (e.g., the convolutions) and all the delays resulting from the processing system and memory management.. In Eq. (9), the processing time is weighted by the probability $p_{u,j}$ that the layer $j$ of CNN $u$ is executed.

(iv) The sink time $t_{f}$ required to transmit the final classification $K_{u,M_{u}}$, for each $u\in\mathbb{N}_{C}$, from the IoT units taking these decisions to the target unit $f$. It is noteworthy to point out that Eq. (10) takes into account all feasible output paths from node $i$ to the target unit $f$, suitably weighted by the probability $g_{u,j}$ that the classification is made at layer $j$ of CNN $u$ in execution on IoT unit $i$.

The constraint in Eq. (4) ensures that each IoT unit contains at most $L$ layers, being $L$ an additional user-defined model hyper-parameter. In particular, when $L=1$, at most one layer can be assigned to an IoT unit, while $L>1$ implies that up to $L$ layers (also belonging to different CNNs) can be assigned to a particular IoT unit. The constraints in Eq. (5) and (6) are meant to take into account the technological limits about memory usage and computational load characterizing each IoT unit. Finally, the constraint in Eq. (7) ensures that each layer $j$, for each $j\in\mathbb{N}_{M}$, is assigned to exactly one node and, at the same time, manages the possibility that the $C$ CNNs might be characterized by a different number $M_{u}\leq M$ of layers, for each $u\in\mathbb{N}_{C}$. In fact, in those cases (i.e., when $M_{u}<M$), the unneeded $\alpha_{u,i,j}$s are set to [math].

When the $j_{1}$-th layer of CNN $u_{1}$ and the $j_{2}$-th layer of CNN $u_{2}$ are shared between the two CNNs, the following constraint is added to the optimization problem

[TABLE]

to ensure that the shared layer is placed on the same IoT unit. In addition, the constraints on the maximum number of layers placed on a IoT unit - Eq. (4) - and the memory usage and computational load constraints - Eqs (5) and (6) - are modified as follows to count the shared layer only once:

[TABLE]

If a layer is shared among $k$ CNNs, the Eqs. (13), (14), and (15) need to take into account $k-1$ out of the $k$ variables corresponding to the shared layer.

The considered class of optimization problems, i.e., the integer quadratic programs, is NP-complete. More specifically, since the $\alpha_{u,i,j}$s are binary variables, it is possible to convert it to a binary linear program, which is one of Karp’s 21 NP-complete problems [42]. In the proposed methodology, the optimization problem is solved through the Gurobi solver555The solver is able to find a solution with a error less than 2% with respect to the optimal placement in less than 2 seconds and a negligible memory occupation in all the considered IoT scenarios described in Section 5..

The optimization problem outcome is the optimal placement $\alpha_{u,i,j}$s of the $C$ CNNs’ layers to the $N$ IoT units minimizing the delay in making a classification. In the event that the optimization provides more than one solution, the optimal placement is any feasible solution with minimal latency. In the rest of this section, the methodology is tailored to three specific configurations of the DL solution, followed by some comments about the open points.

4.1 The configuration with a single CNN

The presence of a single CNN ($C=1$) allows us to rely on N$$M binary variables $\alpha_{i,j}$, determining whether layer $j$ of the CNN is assigned to unit $i$ of the IoT system.

[TABLE]

for each $i\in\mathbb{N}_{N}$ and $j\in\mathbb{N}_{M}$. The objective function in Eq. (3) modelling the latency in making a decision to be minimized is reformulated as:

[TABLE]

Then, the constraints in Eqs. (4), (5), (6) and (7) become:

[TABLE]

while

[TABLE]

account for the transmission time between the source $s$ and the IoT unit running the first layer of the CNN, the processing time on all unit $i$s, and the transmission time between the unit running the $M$-th layer of the CNN and the sink $f$, respectively.

In this configuration, the methodology has been applied to the 5-layer CNN described in Figure 1a, characterized by $M=5$ layers and whose details are in Table IIa. The considered IoT system is the one described in Figure 1b comprising $N=11$ IoT units and being $s$ and $f$ the same unit. The IoT units belong to two different technological families, i.e., STM32H7 (round nodes) and Raspberry Pi 3B+ (squared nodes), whose memory $\bar{m}_{i}$ and computational $\bar{c}_{i}$ constraints are detailed in Table III. An example of the optimization problem outcome in this technological scenario with $L=1$ is given in Figure 1c, whose corresponding $\alpha_{i,j}$s are detailed in Figure 1d. In the optimal placement, involving three STM32H7 units (nodes $n_{05}$, $n_{01}$, and $n_{06}$) and two Raspberry Pi 3B+ (nodes $n_{10}$, and $n_{04}$), the layer $L3$ of the CNN has been assigned to a Raspberry Pi 3B+ IoT unit (i.e., $n_{04}$) since its execution on STM32H7 would violate the memory constraint.

4.2 The configuration with a single early-exit CNN

This configuration refers to the case where a single EX-CNN ($C=1$) has to be placed on the IoT system. Here, the $p_{u,j}$s and $g_{u,j}$s are simplified as $p_{j}$ and $g_{j}$, for each $j\in\mathbb{N}_{M}$, defining the probabilities that layer $j$ is executed and that the final classification is made at layer $j$ (i.e., the direct path from $j$ to the sink is traversed), respectively.

More specifically, the problem variables are simplified into N$$M binary variables $\alpha_{i,j}$ defined in Eq. (16). Instead, the $p_{u,j}$s and $g_{u,j}$s are simplified as $p_{j}$ and $g_{j}$, for each $j\in\mathbb{N}_{M}$, as detailed in Section 4.2.

The objective function modelling the latency in decision making defined in Eq (3) is tailored as follow:

[TABLE]

with constraints as in Eqs. (18), (19), (20), and (21), and where the source time $t_{s}$, the processing time $t_{i}^{(p)}$, and the sink time $t_{f}$ have been modified as follows:

[TABLE]

A 6-layer EX-CNN is shown, as an example, in Figure 2a and detailed in Section 5.1–Table IIa, where $M=6$ and the Early Exit is at layer $j=2$, with a probability $\nu=0.99$ of taking the final classification. Hence, $p_{1}=p_{2}=1$ and $p_{3}=p_{4}=p_{5}=0.01$, with the $g_{j}$s different from zero only at the Early-Exit ($g_{2}=0.99$) and the last layer ($g_{6}=0.01$). In Figure 2, the proposed methodology is applied to this CNN and the IoT system already described in Figure 1b. The methodology outcome is particularly interesting showing that the Early-Exit layer ($j=2$), being particularly demanding in terms of memory, is assigned to the Raspberry Pi 3B+ $n_{04}$ unit. When enough confidence is achieved at Early-Exit ($j=2$), the decision is directly sent from $n_{04}$ to the sink $f$ through $n_{06}$. Otherwise, the processing is forwarded from $n_{04}$ to $n_{01}$ to complete the processing up to $n_{08}$.

4.3 The configuration with multiple CNNs

An interesting application scenario is the one with multiple CNNs, either without or with shared processing layers. EX-CNNs have not been considered here, hence $p_{u,j}=1$, for each $u\in\mathbb{N}_{C}$ and $j\in\{1,\ldots,M_{u}\}$.

In this configuration, the objective function modelling the latency in decision making becomes:

[TABLE]

with constraints defined in Eqs. (4), (5), (6) and (7), and where the source time $t_{s}$, the processing time $t_{i}^{(p)}$, and the sink time $t_{f}$ are modified as follow:

[TABLE]

Finally, to deal with shared layers, the constraint defined in Eq. (12) is introduced per each shared layer, whereas the constraints in Eqs. (4), (5), and (6) are modified accordingly, as detailed in Section 4 with Eqs. (13), (14), and (15). As an example, it is provided the complete extension to deal with the two first shared layers for the case shown in Figure 3b. The additional constraints are defined in Eqs. (38) and (39) to ensure that the shared layers $j=1$ and $j=2$ (of CNNs $u=1$ and $u=2$) are assigned to the same node $i$. Then, the Eqs. (4), (5), and (6) are modified as follow:

[TABLE]

where

[TABLE]

It is noteworthy to point out that there is no difference in defining Eqs. (33), (34), and (35) with the variables of the CNN $u=2$, instead of those of CNN $u=1$, as done here. It is indeed sufficient to relax the constraints with $k-1$ variables out of $k$, where $k$ represents the number of CNNs a layer is shared among, in order to count that shared layer only once.

The proposed methodology has been applied to two instances of the 5-layer CNN described in Figure 1a without common processing layers, operating in the IoT system depicted in Figure 1b and with $L=1$. Interestingly, the outcome of the methodology, depicted in Figure 3a, shows that the placement of both CNNs represent the optimal solution of the single-CNN configuration. Moreover, the methodology has been applied to the case where the convolutional layers $L1$ and $L2$ are shared between the two CNNs. This solution is inspired by the transfer learning paradigm where two CNNs might share low-level representation processing layers, while high-level ones are specific for each CNN. As described in Section 4, the following constraints need to be added to the optimization problem:

[TABLE]

and the constraints Eq. (13), (14) and (15) have to be redefined accordingly. The methodology outcome in this scenario is interesting showing that common layers $L1$ and $L2$ have been placed in IoT units $n_{06}$ and $n_{01}$, respectively, while, after $n_{01}$, the processing takes two different paths.

4.4 Open Points

Currently, the proposed methodology neither takes into account the energy status of IoT units nor network failures [43]. The energy status can be managed by introducing a constraint for each IoT unit depending on the remaining energy value, e.g., forcing the variables $\alpha_{\cdot,i,\cdot}$s of a low-energy node $i$ to zero or introducing penalties for assigning a layer to those nodes. The network failures could be managed by modifying the transmission time defined in Eq. (11) as $(1+\xi^{i,j})\cdot K_{u,j}/\rho\cdot d_{i,k}$ taking into account a probability of retransmission $\xi^{i,j}$ for each pair of nodes $(i,j)$.

It is noteworthy to point out that, thanks to the transfer learning paradigm, the hierarchy of layers of the CNNs can be considered as general feature extractors [44]. For this reason, the deployed CNNs can be easily reconfigured to address a different image-classification problem by replacing only upper layers. Moreover, this optimization phase can be scheduled periodically or when needed to manage variations in the IoT network configuration (e.g., due to the removal or insertion of IoT units). This is a crucial aspect in the scenario of mobile IoT units, a case that is not considered in this paper. In fact, thanks to the transfer learning approach and by periodically recomputing the CNN allocation, the methodology could be applied to units changing their position in the environments they are operating in.

5 Experimental Results

The proposed methodology has been validated on five CNNs and four families of off-the-shelf IoT devices in a synthetic scenario of distributed image classification for the control of a critical area (e.g., recognition of the presence of target objects in a given area through image classification). The monitored area is assumed to be a 30m square and the positions of the IoT units as well as those of the sources $s_{u}$s and the sink $f$ are randomly selected following a uniform distribution. The hyper-parameter $L$, setting the maximum number of CNN layers per IoT unit, ranges from 1 to 5.

The rest of the section is organized as follows. Section 5.1 details the considered CNNs, Section 5.2 describes the families of considered off-the-shelf IoT units and their transmission technologies, and Section 5.3 describes the figure of merit. Sections 5.4 and 5.5 describe the experimental results in two different IoT systems.

5.1 The considered CNNs

The first two CNNs are the 5-layer CNN and the 6-layer EX-CNN shown in Figures 1a and 2a, respectively. These two CNNs, whose values of $m_{j}$s, $c_{j}$s, and $K_{j}$s are detailed in Table IIa, receive in input a 28x28 RGB image and have the following processing layers: two convolutional (with 64 5x5 filters) followed by a 2x2 maximum pooling with stride 2, and three fully-connected layers with 384, 192 and 10 neurons, respectively. In the 6-layer EX-CNN, the Early-Exit is after the first pooling layer and is composed by three fully-connected layers with 384, 192 and 10 neurons.

The third CNN is the AlexNet [16], whose details are given in Table IIb. Such CNN works on 227x227 RGB images and is endowed with 5 convolutional layers (with 96 11x11, 256 5x5, 384 3x3, 384 3x3 and 256 3x3 filters, respectively) and 3 fully-connected layers with 4096, 4096 and 2 neurons. In addition, 3x3 maximum pooling layers with stride 2 are present after the first, second and fifth convolutional layers. An Early-Exit variant of the AlexNet [19] has been considered where the Early-Exit is placed after the second maximum pooling layer and is composed of three fully-connected layers with 128, 64 and 2 neurons, respectively.

The fifth considered CNN is the ResNet 101 [4], whose details are given in Table I. This CNN works on 224x224 RGB images and is composed by a sequence of Bottleneck blocks (i.e., a sequence of a 1x1 convolution, a 3x3 convolution, and a 1x1 convolution with 4 times filters w.r.t to the other two convolutions) of increasing number of filters. The last two layers are an average pooling and a fully-connected.

In all the considered CNNs, the ReLUs, the batch normalization, and the softmax layers have not been explicitly mentioned since they have no parameter to store and negligible computational demands.

5.2 The considered IoT Units

In this experimental section we considered four families of IoT units, whose technological details are given in Table III: the STM32H7, a simple IoT unit endowed with a 400 MHz-Cortex M7 and 1 MB of RAM; the Raspberry Pi 3B+, a more powerful IoT units endowed with a 1.4GHz 64-bit quad-core processor and 1GB of RAM; the OrangePi Zero, endowed with a 800MHz to 1.2GHz quad-core Cortex-A7 and 256 MB of RAM; and the BeagleBone AI having a 1.5GHz dual-core ARM Cortex–A15 and 1GB of RAM.

The maximum memory usage $\bar{m}_{i}$s has been defined as half of the available RAM memory, i.e., $512$KB for the STM32H7, $128$MB for the OrangePi Zero, and $512$MB for the both the BeagleBone AI and the Raspberry Pi3B+. The number of multiplications per second $e_{i}$s has been defined as a tenth of the clock cycles (per number of cores). Hence, $e=40$ for the STM32H7, $e=300$ for the BeagleBone AI ($150$ per core), $e=480$ for the OrangePi Zero ($120$ per core, if we consider the maximum frequency of 1.2GHz), and $e=560$ for the Raspberry 3B+ ($140$ per core). The constraints on the computational load $\bar{c}_{i}$s have not been considered since they are application-specific.

The transmission technologies the IoT units are equipped with are the Wi-Fi 4 (standard IEEE 802.11n) and Wi-Fi HaLow (standard IEEE 802.11n). The transmission range is $d_{t}=7.5m$ (a tenth of the minimum indoor range). The Wi-Fi 4 data-rate is $\rho=72.2$ Mb/s, that corresponds to the single-antenna scenario with 64-QAM modulation on the 20 MHz channel [45], whereas the Wi-Fi HaLow one is $\rho=7.2$ Mb/s with a single-antenna and 64-QAM modulation on the 2 MHz channel [46].

5.3 Figures of Merit

The proposed methodology is evaluated on the “data production to decision making”-latency $t$ defined as the time between image acquisition and classification reception at $f$. To further clarify the effects of data communication and computation, $t$ is split into the transmission $t_{t}$ and the processing $t_{p}$ terms. The former term refers to the sum of all transmission times (from a source to IoT units, between IoT units, or from IoT units to the target unit $f$); the latter sums the processing times on the IoT units. These terms are computed as defined in Section 4, whereas additional sources of delays, such as transmission handshakes or repeated transmissions (due to failures) have been neglected. For each setting, transmission technology, and configuration, the evaluated figure of metric is the mean $\pm$ standard deviation of each latency term, i.e., $t$, $t_{t}$, and $t_{p}$, computed on $500$ randomly generated IoT systems. It is noteworthy to point out that the accuracy has not been considered as a metric since the proposed method does not introduce any approximation w.r.t the original CNN, hence there is no accuracy loss due to placement of the CNN layers.

5.4 The First IoT System: 30 IoT Units and Two Technological Families

The first IoT system comprises $N=30$ IoT units belonging to two technological families, i.e., the STM32H7 and the Raspberry Pi 3B+, with three settings for the IoT units partitioning, i.e., 10%–90%, 50%–50% and 90%–10%.

5.4.1 Single-CNN Configuration

This configuration encompasses a single CNN, either with or without an Early-Exit, to be placed on the considered IoT system. The methodology is tested with $L\in\{1,2,3,4\}$ and compared to the Cloud approximation ($L=5$, referred to as $L=C$), where all the computation can be placed on a single node, i.e., it can be seen as an approximation of sending data directly to the Cloud and then receiving back the result.

The results are shown in Table IVb, in the partition setting 50%–50%. Interesting results arise. First of all, the expected processing time $t_{p}$ is significantly reduced when the Early Exit is employed, as expected and studied in [18, 19] for both the considered CNNs666The latency $t$ and its terms $t_{t}$ and $t_{p}$, are defined as an expected value with EX-CNNs, by weighting their values up to each layer $j$ of CNN $u$ with the probability $g_{u,j}$ of providing the classification.. In the case of 6-layer EX-CNN, the Early-Exit allows to save about 75-84% ($45$ to $80\%$ with Wi-Fi HaLoW) of the latency $t$, whereas on the AlexNet 34 to 50%. After that, with Wi-Fi 4 transmission technology the transmission latency $t_{t}$ is significantly lower than the processing one $t_{p}$, thus it is reasonable to assume that the achieved $t_{p}$ is the minimum feasible in this IoT system. However, with Wi-Fi HaLow, the minimum experimental latency ($t_{p}=44.93$ms for the 5-layer CNN and $t_{p}=1257.71$ms for the AlexNet) cannot be achieved and, in particular, the AlexNet processing with $L=1$ requires more than $3$ seconds (instead of $1.46$s with the Wi-Fi 4 transmission technology), a latency that might be unfeasible in many real applications. The third crucial comment is about the $L=C$ case. The latency $t$ of this case and those of corresponding ones having $L\geq 2$ ($L>2$ with Wi-Fi HaLow) are almost equal (10% increment with $L=1$)and Wi-Fi 4), showing the capability of the proposed methodology of distributing the CNN computation among units with negligible latency increments.

5.4.2 Multi-CNN Configuration

In this configuration, two 5-layer CNNs have to be placed on the considered IoT system, with and without the first two layers shared. Results are presented in Table Vb, for all the configurations and transmission technologies. Several comments arise. At first, in the configurations 90%–10%, the methodology has to often rely on STM32H7 nodes, hence the processing time is significantly increased (the computation capability $e$ of a Raspberry is 14 times greater than that of an STM32H7). This result is even more evident when there are shared layers since the methodology can place less computation on STM32H7s.

The Wi-Fi 4 guarantees transmission latencies $t_{t}$s negligible w.r.t. the processing time $t_{p}$, that represents more than 85% of latency $t$ (up to 96-99% with $L\geq 2$). Interestingly, the processing time is always equal to $89.9$ms, that is the experimental minimum achievable value in this IoT system. This consideration is no longer valid with the Wi-Fi HaLow, where the two terms are comparable, especially when $L=1$. The methodology cannot indeed always achieve the minimum experimental processing latency, but sometimes it has to rely on nearby STM32H7 units, as highlighted by the non-null standard deviation of the $t_{p}$, in the configurations with at least 50% of Raspberry Pi 3B+ units. Interestingly, despite the data-rate of the Wi-Fi 4 is ten times greater than that of Wi-Fi HaLow, the latency $t$ in the harsher case with 90% of STM32H7 units is similar for both the transmission technologies, with a maximum increment of 20% with $L=1$ and no shared layers.

Finally, in all cases with $L\geq 2$ ($L>2$ with Wi-Fi HaLow) the total latency $t$ is comparable to the case ($L=5$), as in Single-CNN configuration (Section 5.4.1). It is crucial to point out the importance of this result because distributing the CNN processing on various IoT units with a negligible increment in latency $t$ will allow defining a pipeline in processing sequence of images. Indeed, when a unit has carried out the processing of CNN layers is designed to and sent the computed representation to the subsequent node, it is ready to operate on the next image, as in processor pipelines. Hence, the throughput of CNN processing can be significantly increased by processing images in a pipeline, with bottleneck the IoT unit responsible for the highest processing time.

5.5 The Second IoT System: 50 IoT Units and Three Technological Families

A second IoT system is considered with $N=50$ units belonging to three different technological families equipped with the Wi-Fi 4 transmission technology and partitioned as follows: 45% of OrangePi Zero, 45% of BeagleBone AI, and 10% of Raspberry Pi 3B+. An example of this IoT system is shown in Figure 4a, where the OrangePi Zero units are represented by a circle, the BeagleBone AI units by a square, the Raspberry Pi 3B+ units by a diamond, the sources by an asterisk, and the target unit by a circled cross.

The scenario is interesting because the most powerful IoT units in terms of both memory and computation capabilities, i.e., the Raspberry Pi 3B+, are just a few (about 5 in each simulated IoT system), whereas the remaining IoT units are characterized by contrasting peculiarities: on one hand the BeagleBone AI units have the same memory capability of the Raspberry Pi 3B+ but only 54% of the computation one; on the other hand the OrangePi Zero have almost the same computation capability of the Raspberry Pi 3B+ (85%, 160% if compared to BeagleBone AI), but only a fourth of the memory capability. It is worth noting that both the Raspberry Pi 3B+ and BeagleBone AI can store all the layers of the considered CNNs, whereas being the OrangePi Zero units with 128 MB of RAM they cannot store all the layers of the ResNet and the (EX–)AlexNet CNNs. As a consequence, balancing between the faster OrangePi Zero units (in terms of computation capability) and the slower but with higher memory capacity BeagleBone AI units is expected in this IoT system (at least after all the Raspberry Pi 3B+ units have been considered, if enough closer).

In addition to the figures of merit presented in Section 5.3, the number of considered nodes $\eta_{x}$ is taken into account, where $x$ is a technological family of IoT units, i.e. $x$ can be $R$, $O$, or $B$, representing the Raspberry Pi 3B+, the OrangePi Zero, and the BeagleBone AI units, respectively.

5.5.1 Single-CNN Configuration

In this configuration, 1 ResNet CNN has to be placed in this IoT system.

The results are shown in Table VI. The processing latency $t_{p}$ of a ResNet CNN is extremely high but proportional to the number of operations required by such CNN to process a single image, i.e., about 7.5 times than those of the AlexNet. The methodology indeed does not encompass any optimization in processing the convolutions or any other optimization unless one re-compute the number of required operations accordingly. Since the latency $t$ is almost only composed by the processing $t_{p}$ (97 to 99% of $t$), the methodology relies only on the IoT units having the highest computation capability, as highlighted by the values of $\eta_{B}$ that are close to zero in almost all the cases.

5.5.2 Multi-CNN Configuration

In this bunch of experiments, 1, 3, or 4 AlexNet CNNs have to be placed in the IoT system. It is worth noting that in the latter case, when $L=1$, 32 nodes out of 50 have to be used, allowing us to in-depth analyze the best placements.

The results are shown in Table VII, with the hyper-parameter $L$ ranging from 1 to 4 and the case $L=C$ simulating the Cloud, i.e., when all the layers of a CNN can be placed on the same IoT unit due to $L=M$777It is worth nothing that with $L=C$, the layers of an (EX–)AlexNet CNN can be placed on a single node if and only if Raspberry Pi 3B+ or BeagleBone AI IoT units are employed. If an OrangePi unit is selected by the methodology, at least another IoT unit has to be considered to place all the layers, also in this configuration.. Several comments arise. First of all, the latency $t$ of 1 AlexNet is higher than that in the IoT system comprising only Raspberry Pi 3B+ and STM32H7 (see Section 5.4): this is justified by the fact that in this IoT system the probability that an IoT unit is a Raspberry is 10%, instead of 50%, and both the OrangePi Zero and the BeagleBone AI units have a smaller computation capability.

Second, the latency $t$ of 3 AlexNet is close to that with 1 AlexNet multiplied by 3. In fact, the difference in percentage ranges from 0.6% to 3.5% ($L=C$ to $L=1$)888The error percentage between the latency $t_{3A}$ of placing 3 AlexNet CNNs and the latency $t_{A}$ of placing 1 AlexNet multiplied by 3 is computed as follows: $(t_{3A}-3\cdot t_{A})/t_{3A}$.. When 4 AlexNet CNNs are employed, the range is slightly higher, i.e., 1 to 5% on the same values of $L$. This result is very interesting, showing the effectiveness of the proposed methodology in placing the CNNs in the given IoT system. Moreover, by observing the values of $\eta_{R}$, $\eta_{O}$, $\eta_{B}$ it is clear that whenever possible the methodology relies on the fastest units, as expected by the fact that the transmission latency $t_{t}$ is, in all the cases, significantly smaller than the processing $t_{p}$ one. Finally, as commented in Section 5.4.2, the latency $t$ with $L=1$ and $L=2$ is close to that with $L=C$, with an increment ranging from 13% to 18%, and from 6% to 9%, with 1 and 4 AlexNet CNNs, respectively, allowing us to define processing pipelines in the considered IoT system, to further reduce the latency $t$.

In Table VIII, the same IoT scenario is investigated with 1, 3, and 4 EX-AlexNet to be placed. The latency $t$ and its components $t_{p}$ and $t_{t}$ are defined as an expected value, by weighting the latency of each possible path within the EX-CNN by its probability. The mean numbers of nodes used $\eta_{R}$, $\eta_{O}$, and $\eta_{B}$ are instead computed on the longest path within the EX-CNN.

The trend in the results is analogous to the case with AlexNet CNNs, with smaller errors. The difference in percentage between placing 3 and 4 EX-AlexNet and 1 EX-AlexNet multiplied by 3 and 4 ranges from 0.2% to 2.5% and from 0.4% to 3.7% ($L=C$ to $L=1$), respectively. Interestingly, the values of $\eta_{B}$ are higher in this group of experiments, showing that the methodology more often relies on closer BeagleBone AI units to place part of the EX-AlexNet computation, reasonably on the less probable path. The OrangePi Zero units are more often used in this scenario as well.

6 Porting a CNN to a real IoT System

The methodology has been also applied to the placement of the 5-layer CNN described in Section 4.1 and depicted in Fig. 1(a) on a real technological scenario comprising two STM32H7s and one Raspberry Pi 3B+. The transmission technology is the Wi-Fi 4 and the connectivity is provided locally by the GL.iNet GL-MT300N-V2 router. The goal of this experiment is to compare the figures of merit $t$, $t_{t}$, and $t_{p}$ of the CNN placement provided by the methodology with those of the real CNN placement in the considered technological scenario. With $L=4$ and the nodes equally spaced (each at distance 1 from each other), the methodology assigned the first four layers of the CNN to the Raspberry and the fifth layer to one of the two STM32H7s. The measured transmission and processing times, shown in Table IX, are particularly interesting, showing that the experimental transmission time $t_{t}$ is almost equal to the methodology estimation, whereas the experimental processing time $t_{p}$ is 30% larger. This is justified by the fact the model considered only the multiplications. In particular, the first four layers on Raspberry Pi 3B+ took 68.29 ms instead of 44.93 ms, whereas the fifth layer on STM32H7 176 $\mu$s (32 $\mu$s with code optimization) instead of 50 $\mu$s.

It is worth nothing that the measured processing time $t_{p}$ of the AlexNet on the Raspberry Pi 3B+ (median over 100 runs) is 1119.47 ms, whereas the one provided by the methodology is 1257.71 ms, showing that the approximation given in Section 4 well describe this technological scenario.

7 Conclusions

The aim of this paper was to introduce a novel effective methodology for the optimal placement of CNNs on IoT systems. Such a methodology is general enough to be applied to multiple sources of data and multiple CNNs operating in the same IoT system and has been formalized as an optimization problem where the latency between image acquisitions and the decision making is minimized. Future works will encompass the extension of the methodology with dynamic scheduling and routing algorithms for IoT to deal with communications failures and mobile units as well as improve the performance by exploiting the resources that are made available by early-exit mechanisms.