Application Specific Compression of Deep Learning Models

TL;DR

This paper introduces Application Specific Compression (ASC), a method that prunes deep learning models based on target application needs, resulting in more efficient and effective models for specific tasks.

Contribution

The paper proposes a novel application-aware pruning technique that improves model performance for specific tasks compared to traditional, application-agnostic compression methods.

Findings

ASC outperforms existing compression methods on BERT models.

Customized models show better accuracy for target applications.

Pruning reduces model size while maintaining or improving task performance.

Abstract

Large Deep Learning models are compressed and deployed for specific applications. However, current Deep Learning model compression methods do not utilize the information about the target application. As a result, the compressed models are application agnostic. Our goal is to customize the model compression process to create a compressed model that will perform better for the target application. Our method, Application Specific Compression (ASC), identifies and prunes components of the large Deep Learning model that are redundant specifically for the given target application. The intuition of our work is to prune the parts of the network that do not contribute significantly to updating the data representation for the given application. We have experimented with the BERT family of models for three applications: Extractive QA, Natural Language Inference, and Paraphrase Identification. We observe that customized compressed models created using ASC method perform better than existing model compression methods and off-the-shelf compressed models.