Rethinking Data Distillation: Do Not Overlook Calibration

TL;DR

This paper investigates the calibration issues of neural networks trained on distilled data, revealing inherent limitations and proposing novel methods to improve calibration without sacrificing efficiency.

Contribution

It identifies why existing calibration methods fail on distilled data and introduces Masked Temperature Scaling and Masked Distillation Training as effective solutions.

Findings

Existing calibration methods fail on distilled data.

Distilled data causes concentrated logits and information loss.

Proposed methods improve calibration while maintaining efficiency.

Abstract

Neural networks trained on distilled data often produce over-confident output and require correction by calibration methods. Existing calibration methods such as temperature scaling and mixup work well for networks trained on original large-scale data. However, we find that these methods fail to calibrate networks trained on data distilled from large source datasets. In this paper, we show that distilled data lead to networks that are not calibratable due to (i) a more concentrated distribution of the maximum logits and (ii) the loss of information that is semantically meaningful but unrelated to classification tasks. To address this problem, we propose Masked Temperature Scaling (MTS) and Masked Distillation Training (MDT) which mitigate the limitations of distilled data and achieve better calibration results while maintaining the efficiency of dataset distillation.