Reducing Inference Latency with Concurrent Architectures for Image Recognition

TL;DR

This paper introduces a new neural architecture search space that relaxes single-chain dependencies to enable higher concurrency, aiming to reduce inference latency in image recognition tasks.

Contribution

It proposes a novel NAS space, a scoring metric for architecture comparison, and a generator with transformation blocks that outperform existing methods.

Findings

New architectures reduce inference latency

Proposed score effectively compares architectures

Generator and transformation blocks improve architecture quality

Abstract

Satisfying the high computation demand of modern deep learning architectures is challenging for achieving low inference latency. The current approaches in decreasing latency only increase parallelism within a layer. This is because architectures typically capture a single-chain dependency pattern that prevents efficient distribution with a higher concurrency (i.e., simultaneous execution of one inference among devices). Such single-chain dependencies are so widespread that even implicitly biases recent neural architecture search (NAS) studies. In this visionary paper, we draw attention to an entirely new space of NAS that relaxes the single-chain dependency to provide higher concurrency and distribution opportunities. To quantitatively compare these architectures, we propose a score that encapsulates crucial metrics such as communication, concurrency, and load balancing. Additionally, we propose a new generator and transformation block that consistently deliver superior architectures compared to current state-of-the-art methods. Finally, our preliminary results show that these new architectures reduce the inference latency and deserve more attention.