Keynote: Small Neural Nets Are Beautiful: Enabling Embedded Systems with Small Deep-Neural-Network Architectures

TL;DR

This paper discusses the development of small, efficient deep neural network architectures suitable for embedded systems, highlighting principles for designing compact models like SqueezeNet that require minimal storage.

Contribution

It introduces a novel neural network architecture, SqueezeNet, and provides a playbook for designing small deep neural networks for embedded applications.

Findings

SqueezeNet requires only 480KB of storage.

Design principles from microprocessor architecture are applicable to neural network design.

A practical playbook for creating small neural networks is proposed.

Abstract

Over the last five years Deep Neural Nets have offered more accurate solutions to many problems in speech recognition, and computer vision, and these solutions have surpassed a threshold of acceptability for many applications. As a result, Deep Neural Networks have supplanted other approaches to solving problems in these areas, and enabled many new applications. While the design of Deep Neural Nets is still something of an art form, in our work we have found basic principles of design space exploration used to develop embedded microprocessor architectures to be highly applicable to the design of Deep Neural Net architectures. In particular, we have used these design principles to create a novel Deep Neural Net called SqueezeNet that requires as little as 480KB of storage for its model parameters. We have further integrated all these experiences to develop something of a playbook for creating small Deep Neural Nets for embedded systems.