Adaptive Dataset Quantization

TL;DR

This paper introduces Adaptive Dataset Quantization (ADQ), a versatile framework for dataset compression that improves upon naive quantization by adaptively sampling data based on importance, leading to better generalization and state-of-the-art results.

Contribution

The paper proposes a novel adaptive sampling strategy for dataset quantization that considers importance, diversity, and representativeness, addressing limitations of existing methods like dataset distillation and coreset selection.

Findings

ADQ outperforms naive dataset quantization by 3% on average across datasets.

The method demonstrates superior generalization across different neural network architectures.

Extensive experiments validate the effectiveness of adaptive sampling based on importance, diversity, and representativeness.

Abstract

Contemporary deep learning, characterized by the training of cumbersome neural networks on massive datasets, confronts substantial computational hurdles. To alleviate heavy data storage burdens on limited hardware resources, numerous dataset compression methods such as dataset distillation (DD) and coreset selection have emerged to obtain a compact but informative dataset through synthesis or selection for efficient training. However, DD involves an expensive optimization procedure and exhibits limited generalization across unseen architectures, while coreset selection is limited by its low data keep ratio and reliance on heuristics, hindering its practicality and feasibility. To address these limitations, we introduce a newly versatile framework for dataset compression, namely Adaptive Dataset Quantization (ADQ). Specifically, we first identify the sub-optimal performance of naive Dataset Quantization (DQ), which relies on uniform sampling and overlooks the varying importance of each generated bin. Subsequently, we propose a novel adaptive sampling strategy through the evaluation of generated bins' representativeness score, diversity score and importance score, where the former two scores are quantified by the texture level and contrastive learning-based techniques, respectively. Extensive experiments demonstrate that our method not only exhibits superior generalization capability across different architectures, but also attains state-of-the-art results, surpassing DQ by average 3\% on various datasets.