Model Complexity of Program Phases

TL;DR

This paper explores the fundamental tradeoff between implementation cost and prediction quality in resource-constrained sequence prediction models, providing theoretical insights and empirical methods for deep neural networks.

Contribution

It formulates a theory and empirical procedure to analyze the cost-quality tradeoff in models for resource-limited tasks, focusing on deep neural networks.

Findings

Tradeoff space characterized for neural network models

Theoretical framework for cost-quality analysis

Empirical methods to explore model performance constraints

Abstract

In resource limited computing systems, sequence prediction models must operate under tight constraints. Various models are available that cater to prediction under these conditions that in some way focus on reducing the cost of implementation. These resource constrained sequence prediction models, in practice, exhibit a fundamental tradeoff between the cost of implementation and the quality of its predictions. This fundamental tradeoff seems to be largely unexplored for models for different tasks. Here we formulate the necessary theory and an associated empirical procedure to explore this tradeoff space for a particular family of machine learning models such as deep neural networks. We anticipate that the knowledge of the behavior of this tradeoff may be beneficial in understanding the theoretical and practical limits of creation and deployment of models for resource constrained tasks.