Variant Parallelism: Lightweight Deep Convolutional Models for Distributed Inference on IoT Devices

TL;DR

This paper introduces variant parallelism, a lightweight ensemble-based method for distributed deep learning inference on IoT devices, reducing communication overhead and improving fault tolerance while maintaining high accuracy.

Contribution

The authors propose variant parallelism, a novel ensemble-based distribution technique that generates lightweight model variants for efficient, fault-tolerant inference on resource-constrained IoT devices.

Findings

Models are 5.8-7.1x smaller in parameters

Achieve 4.3-31x fewer MACs and 2.5-13.2x faster response time

Maintain or improve accuracy compared to MobileNetV2

Abstract

Two major techniques are commonly used to meet real-time inference limitations when distributing models across resource-constrained IoT devices: (1) model parallelism (MP) and (2) class parallelism (CP). In MP, transmitting bulky intermediate data (orders of magnitude larger than input) between devices imposes huge communication overhead. Although CP solves this problem, it has limitations on the number of sub-models. In addition, both solutions are fault intolerant, an issue when deployed on edge devices. We propose variant parallelism (VP), an ensemble-based deep learning distribution method where different variants of a main model are generated and can be deployed on separate machines. We design a family of lighter models around the original model, and train them simultaneously to improve accuracy over single models. Our experimental results on six common mid-sized object recognition datasets demonstrate that our models can have 5.8-7.1x fewer parameters, 4.3-31x fewer multiply-accumulations (MACs), and 2.5-13.2x less response time on atomic inputs compared to MobileNetV2 while achieving comparable or higher accuracy. Our technique easily generates several variants of the base architecture. Each variant returns only 2k outputs 1 <= k <= (#classes/2), representing Top-k classes, instead of tons of floating point values required in MP. Since each variant provides a full-class prediction, our approach maintains higher availability compared with MP and CP in presence of failure.