Towards Trustworthy Dataset Distillation

TL;DR

This paper introduces TrustDD, a new dataset distillation approach that creates compact datasets capable of training models for both in-distribution classification and out-of-distribution detection, enhancing efficiency and trustworthiness.

Contribution

The paper proposes TrustDD, a novel dataset distillation paradigm that jointly distills in-distribution data and outliers, and introduces Pseudo-Outlier Exposure to generate pseudo-outliers without real outlier data.

Findings

TrustDD improves both InD classification and OOD detection performance.

POE surpasses the state-of-the-art Outlier Exposure method.

TrustDD is more trustworthy and suitable for open-world scenarios.

Abstract

Efficiency and trustworthiness are two eternal pursuits when applying deep learning in real-world applications. With regard to efficiency, dataset distillation (DD) endeavors to reduce training costs by distilling the large dataset into a tiny synthetic dataset. However, existing methods merely concentrate on in-distribution (InD) classification in a closed-world setting, disregarding out-of-distribution (OOD) samples. On the other hand, OOD detection aims to enhance models' trustworthiness, which is always inefficiently achieved in full-data settings. For the first time, we simultaneously consider both issues and propose a novel paradigm called Trustworthy Dataset Distillation (TrustDD). By distilling both InD samples and outliers, the condensed datasets are capable of training models competent in both InD classification and OOD detection. To alleviate the requirement of real outlier data, we further propose to corrupt InD samples to generate pseudo-outliers, namely Pseudo-Outlier Exposure (POE). Comprehensive experiments on various settings demonstrate the effectiveness of TrustDD, and POE surpasses the state-of-the-art method Outlier Exposure (OE). Compared with the preceding DD, TrustDD is more trustworthy and applicable to open-world scenarios. Our code is available at https://github.com/mashijie1028/TrustDD