Compressing Model with Few Class-Imbalance Samples: An Out-of-Distribution Expedition

TL;DR

This paper introduces OE-FSMC, a novel framework that uses out-of-distribution data to improve few-sample model compression under class imbalance, enhancing accuracy in limited-data scenarios.

Contribution

The paper proposes a new adaptive framework that incorporates OOD data into compression and fine-tuning to address class imbalance in few-sample model compression.

Findings

OE-FSMC effectively mitigates accuracy loss due to class imbalance.

The framework improves performance across multiple benchmark datasets.

It can be integrated with existing compression methods seamlessly.

Abstract

In recent years, as a compromise between privacy and performance, few-sample model compression has been widely adopted to deal with limited data resulting from privacy and security concerns. However, when the number of available samples is extremely limited, class imbalance becomes a common and tricky problem. Achieving an equal number of samples across all classes is often costly and impractical in real-world applications, and previous studies on few-sample model compression have mostly ignored this significant issue. Our experiments comprehensively demonstrate that class imbalance negatively affects the overall performance of few-sample model compression methods. To address this problem, we propose a novel and adaptive framework named OOD-Enhanced Few-Sample Model Compression (OE-FSMC). This framework integrates easily accessible out-of-distribution (OOD) data into both the compression and fine-tuning processes, effectively rebalancing the training distribution. We also incorporate a joint distillation loss and a regularization term to reduce the risk of the model overfitting to the OOD data. Extensive experiments on multiple benchmark datasets show that our framework can be seamlessly incorporated into existing few-sample model compression methods, effectively mitigating the accuracy degradation caused by class imbalance.