TOCO: A Framework for Compressing Neural Network Models Based on Tolerance Analysis

TL;DR

TOCO is a flexible framework for neural network model compression that uses tolerance analysis to maintain accuracy and adapt to various hardware constraints efficiently.

Contribution

It introduces a novel tolerance-based analysis method that enables fine-grained, hardware-agnostic compression of neural networks, improving over heuristic approaches.

Findings

Achieves accurate model compression with minimal accuracy loss.

Supports diverse hardware constraints through decoupled analysis and compression.

Demonstrates effectiveness on multiple neural network architectures.

Abstract

Neural network compression methods have enabled deploying large models on emerging edge devices with little cost, by adapting already-trained models to the constraints of these devices. The rapid development of AI-capable edge devices with limited computation and storage requires streamlined methodologies that can efficiently satisfy the constraints of different devices. In contrast, existing methods often rely on heuristic and manual adjustments to maintain accuracy, support only coarse compression policies, or target specific device constraints that limit their applicability. We address these limitations by proposing the TOlerance-based COmpression (TOCO) framework. TOCO uses an in-depth analysis of the model, to maintain the accuracy, in an active learning system. The results of the analysis are tolerances that can be used to perform compression in a fine-grained manner. Finally, by decoupling compression from the tolerance analysis, TOCO allows flexibility to changes in the hardware.