Out-of-Distribution Robustness in Deep Learning Compression

TL;DR

This paper explores methods to improve the robustness of deep neural network compression systems against out-of-distribution data and distribution shifts, proposing algorithms based on distributionally-robust optimization and structured latent codes.

Contribution

It introduces the first study of OOD robust compression, proposing two frameworks that enhance robustness and analyzing their tradeoffs and theoretical properties.

Findings

Both proposed methods improve robustness over standard compressors.

Structured latent codes can outperform distributionally-robust optimization.

Tradeoffs exist between robustness and compression distortion.

Abstract

In recent years, deep neural network (DNN) compression systems have proved to be highly effective for designing source codes for many natural sources. However, like many other machine learning systems, these compressors suffer from vulnerabilities to distribution shifts as well as out-of-distribution (OOD) data, which reduces their real-world applications. In this paper, we initiate the study of OOD robust compression. Considering robustness to two types of ambiguity sets (Wasserstein balls and group shifts), we propose algorithmic and architectural frameworks built on two principled methods: one that trains DNN compressors using distributionally-robust optimization (DRO), and the other which uses a structured latent code. Our results demonstrate that both methods enforce robustness compared to a standard DNN compressor, and that using a structured code can be superior to the DRO compressor. We observe tradeoffs between robustness and distortion and corroborate these findings theoretically for a specific class of sources.