CEKD:Cross Ensemble Knowledge Distillation for Augmented Fine-grained Data

TL;DR

This paper introduces CEKD, a novel cross ensemble knowledge distillation approach that improves fine-grained image classification by reducing noise and target conflicts, achieving state-of-the-art results with only image-level supervision.

Contribution

The paper proposes a new cross ensemble knowledge distillation framework with a cross distillation module and a collaborative ensemble module for fine-grained learning.

Findings

CEKD outperforms state-of-the-art methods on multiple benchmarks.

The model achieves high accuracy with only image-level labels.

Extensive experiments validate the effectiveness of the proposed approach.

Abstract

Data augmentation has been proved effective in training deep models. Existing data augmentation methods tackle the fine-grained problem by blending image pairs and fusing corresponding labels according to the statistics of mixed pixels, which produces additional noise harmful to the performance of networks. Motivated by this, we present a simple yet effective cross ensemble knowledge distillation (CEKD) model for fine-grained feature learning. We innovatively propose a cross distillation module to provide additional supervision to alleviate the noise problem, and propose a collaborative ensemble module to overcome the target conflict problem. The proposed model can be trained in an end-to-end manner, and only requires image-level label supervision. Extensive experiments on widely used fine-grained benchmarks demonstrate the effectiveness of our proposed model. Specifically, with the backbone of ResNet-101, CEKD obtains the accuracy of 89.59%, 95.96% and 94.56% in three datasets respectively, outperforming state-of-the-art API-Net by 0.99%, 1.06% and 1.16%.