CDFKD-MFS: Collaborative Data-free Knowledge Distillation via Multi-level Feature Sharing

TL;DR

This paper introduces CDFKD-MFS, a novel framework for compressing multiple neural networks into a smaller model without using original data, leveraging multi-level feature sharing and adversarial training to improve performance and address privacy concerns.

Contribution

The paper proposes a new data-free knowledge distillation framework that enables multi-model compression using multi-level feature sharing and adversarial training, with adaptive prediction aggregation.

Findings

Achieves higher accuracy than existing methods on CIFAR-100, Caltech-101, and mini-ImageNet datasets.

Effectively compresses multiple models into a smaller model without access to original data.

Demonstrates the effectiveness of multi-level feature sharing and attention-based aggregation.

Abstract

Recently, the compression and deployment of powerful deep neural networks (DNNs) on resource-limited edge devices to provide intelligent services have become attractive tasks. Although knowledge distillation (KD) is a feasible solution for compression, its requirement on the original dataset raises privacy concerns. In addition, it is common to integrate multiple pretrained models to achieve satisfactory performance. How to compress multiple models into a tiny model is challenging, especially when the original data are unavailable. To tackle this challenge, we propose a framework termed collaborative data-free knowledge distillation via multi-level feature sharing (CDFKD-MFS), which consists of a multi-header student module, an asymmetric adversarial data-free KD module, and an attention-based aggregation module. In this framework, the student model equipped with a multi-level feature-sharing structure learns from multiple teacher models and is trained together with a generator in an asymmetric adversarial manner. When some real samples are available, the attention module adaptively aggregates predictions of the student headers, which can further improve performance. We conduct extensive experiments on three popular computer visual datasets. In particular, compared with the most competitive alternative, the accuracy of the proposed framework is 1.18\% higher on the CIFAR-100 dataset, 1.67\% higher on the Caltech-101 dataset, and 2.99\% higher on the mini-ImageNet dataset.