Safety and Performance, Why not Both? Bi-Objective Optimized Model Compression toward AI Software Deployment

TL;DR

This paper introduces SafeCompress, a test-driven sparse training framework for safe model compression that balances AI model size reduction with robustness against attacks like membership inference.

Contribution

It proposes a novel safety-performance co-optimization approach for model compression, integrating attack simulation into the training process to enhance security.

Findings

Effective compression while maintaining model safety.

Generalizes to multiple attack types beyond MIA.

Validated on diverse datasets for vision and NLP.

Abstract

The size of deep learning models in artificial intelligence (AI) software is increasing rapidly, which hinders the large-scale deployment on resource-restricted devices (e.g., smartphones). To mitigate this issue, AI software compression plays a crucial role, which aims to compress model size while keeping high performance. However, the intrinsic defects in the big model may be inherited by the compressed one. Such defects may be easily leveraged by attackers, since the compressed models are usually deployed in a large number of devices without adequate protection. In this paper, we try to address the safe model compression problem from a safety-performance co-optimization perspective. Specifically, inspired by the test-driven development (TDD) paradigm in software engineering, we propose a test-driven sparse training framework called SafeCompress. By simulating the attack mechanism as the safety test, SafeCompress can automatically compress a big model to a small one following the dynamic sparse training paradigm. Further, considering a representative attack, i.e., membership inference attack (MIA), we develop a concrete safe model compression mechanism, called MIA-SafeCompress. Extensive experiments are conducted to evaluate MIA-SafeCompress on five datasets for both computer vision and natural language processing tasks. The results verify the effectiveness and generalization of our method. We also discuss how to adapt SafeCompress to other attacks besides MIA, demonstrating the flexibility of SafeCompress.