Tuning of Mixture-of-Experts Mixed-Precision Neural Networks

TL;DR

This paper introduces mixed-precision techniques and a mixture-of-experts approach in neural networks to reduce memory usage and boost inference speed on commodity hardware, making deep learning more accessible.

Contribution

It adds new data types to Caffe for mixed-precision inference and proposes a variation of mixture-of-experts to improve speed on AlexNet, with open-source implementation.

Findings

Memory usage decreased up to 3.29x

Inference speed increased up to 3.01x

Techniques applicable to various machine learning problems

Abstract

Deep learning has become a useful data analysis method, however mainstream adaption in distributed computer software and embedded devices has been low so far. Often, adding deep learning inference in mainstream applications and devices requires new hardware with signal processors suited for convolutional neural networks. This work adds new data types (quantized 16-bit and 8-bit integer, 16-bit floating point) to Caffe in order to save memory and increase inference speed on existing commodity graphics processors with OpenCL, common in everyday devices. Existing models can be executed effortlessly in mixed-precision mode. Additionally, we propose a variation of mixture-of-experts to increase inference speed on AlexNet for image classification. We managed to decrease memory usage up to 3.29x while increasing inference speed up to 3.01x on certain devices. We demonstrate with five simple examples how the presented techniques can easily be applied to different machine learning problems. The whole pipeline, consisting of models, example python scripts and modified Caffe library, is available as Open Source software.