Adaptive Data Dropout: Towards Self-Regulated Learning in Deep Neural Networks

TL;DR

This paper introduces Adaptive Data Dropout, a dynamic data selection method for deep neural network training that adjusts data exposure based on performance feedback to improve efficiency and robustness.

Contribution

It proposes a novel self-regulated learning framework that adaptively modulates training data exposure during training, unlike fixed schedule methods.

Findings

Reduces effective training steps while maintaining accuracy.

Balances exploration and consolidation through online data modulation.

Demonstrates improved efficiency on image classification benchmarks.

Abstract

Deep neural networks are typically trained by uniformly sampling large datasets across epochs, despite evidence that not all samples contribute equally throughout learning. Recent work shows that progressively reducing the amount of training data can improve efficiency and generalization, but existing methods rely on fixed schedules that do not adapt during training. In this work, we propose Adaptive Data Dropout, a simple framework that dynamically adjusts the subset of training data based on performance feedback. Inspired by self-regulated learning, our approach treats data selection as an adaptive process, increasing or decreasing data exposure in response to changes in training accuracy. We introduce a lightweight stochastic update mechanism that modulates the dropout schedule online, allowing the model to balance exploration and consolidation over time. Experiments on standard image classification benchmarks show that our method reduces effective training steps while maintaining competitive accuracy compared to static data dropout strategies. These results highlight adaptive data selection as a promising direction for efficient and robust training. Code will be released.