Efficient model compression with Random Operation Access Specific Tile (ROAST) hashing

TL;DR

This paper introduces ROAST hashing, a cache-efficient, model-agnostic compression method that significantly reduces model size and training/inference time, enabling deployment of large models like BERT on edge devices without quality loss.

Contribution

ROAST hashing is a novel, cache-friendly model compression technique that outperforms existing methods and enables the first compressed BERT suitable for resource-constrained devices.

Findings

ROAST is up to 25x faster to train and 50x faster to infer than HashedNet.

ROAST achieves 100x to 1000x compression of BERT without quality degradation.

Global weight sharing in ROAST is empirically and theoretically superior to local sharing.

Abstract

Advancements in deep learning are often associated with increasing model sizes. The model size dramatically affects the deployment cost and latency of deep models. For instance, models like BERT cannot be deployed on edge devices and mobiles due to their sheer size. As a result, most advances in Deep Learning are yet to reach the edge. Model compression has sought much-deserved attention in literature across natural language processing, vision, and recommendation domains. This paper proposes a model-agnostic, cache-friendly model compression approach: Random Operation Access Specific Tile (ROAST) hashing. ROAST collapses the parameters by clubbing them through a lightweight mapping. Notably, while clubbing these parameters, ROAST utilizes cache hierarchies by aligning the memory access pattern with the parameter access pattern. ROAST is up to $\sim 25 \times$ faster to train and $\sim 50 \times$ faster to infer than the popular parameter sharing method HashedNet. Additionally, ROAST introduces global weight sharing, which is empirically and theoretically superior to local weight sharing in HashedNet, and can be of independent interest in itself. With ROAST, we present the first compressed BERT, which is $100\times - 1000\times$ smaller but does not result in quality degradation. These compression levels on universal architecture like transformers are promising for the future of SOTA model deployment on resource-constrained devices like mobile and edge devices