Heterogeneous Dual-Core Overlay Processor for Light-Weight CNNs

TL;DR

This paper introduces a heterogeneous dual-core processor architecture optimized for light-weight CNNs, significantly improving runtime efficiency and throughput by leveraging specialized cores and a novel scheduling algorithm.

Contribution

It proposes the first in-depth heterogeneous dual-core architecture for light-weight CNNs, optimizing cores for different layer types and developing a scheduling algorithm for enhanced parallelism.

Findings

11% improvement in runtime PE efficiency

31% increase in throughput for single network

11% throughput gain for multiple networks

Abstract

Light-weight convolutional neural networks (CNNs) have small complexity and are good candidates for low-power, high-throughput inference. Such networks are heterogeneous in terms of computation-to-communication (CTC) ratios and computation patterns between layers, especially for different layer types. Yet, existing AI processors either use homogeneous processing elements (PEs), resulting in low runtime PE efficiency, or run different layers on heterogeneous PEs in sequential, introducing resource redundancy. This paper proposes a heterogeneous dual-core architecture (dual-OPU), where one core is optimized for regular convolution layers and the other for depthwise convolution layers. PEs are homogeneous with each core. To make full use of dual-core parallelism, we develop a scheduling algorithm to concurrently execute layers for different input images on dual-core and balance parallel workload. Meanwhile, we automatically tune the PE number for a core and tune the input size for each PE to maximize throughput. Compared with a single-core processor with the same area for a single network, heterogeneous dual-OPU on average improves runtime PE efficiency and throughput by 11% and 31%, respectively. For a workload of multiple networks, dual-OPU improves average throughput by 11% compared with the state-of-the-art processors scaled to the same area. To the best of our knowledge, it is the first in-depth study on the heterogeneous dual-core processor for light-weight CNNs.