X-model: Improving Data Efficiency in Deep Learning with A Minimax Model

TL;DR

This paper introduces the X-model, a novel deep learning approach that enhances data efficiency for both classification and regression tasks by combining invariance to data and model stochasticity through a minimax game.

Contribution

The X-model is the first to simultaneously encourage invariance to data and model stochasticity in deep regression, improving data efficiency across diverse tasks.

Findings

Outperforms existing methods in age estimation and keypoint localization.

Effective across synthetic and real datasets in 2D and 3D.

Enhances data efficiency in multi-category object recognition.

Abstract

To mitigate the burden of data labeling, we aim at improving data efficiency for both classification and regression setups in deep learning. However, the current focus is on classification problems while rare attention has been paid to deep regression, which usually requires more human effort to labeling. Further, due to the intrinsic difference between categorical and continuous label space, the common intuitions for classification, e.g., cluster assumptions or pseudo labeling strategies, cannot be naturally adapted into deep regression. To this end, we first delved into the existing data-efficient methods in deep learning and found that they either encourage invariance to data stochasticity (e.g., consistency regularization under different augmentations) or model stochasticity (e.g., difference penalty for predictions of models with different dropout). To take the power of both worlds, we propose a novel X-model by simultaneously encouraging the invariance to {data stochasticity} and {model stochasticity}. Further, the X-model plays a minimax game between the feature extractor and task-specific heads to further enhance the invariance to model stochasticity. Extensive experiments verify the superiority of the X-model among various tasks, from a single-value prediction task of age estimation to a dense-value prediction task of keypoint localization, a 2D synthetic, and a 3D realistic dataset, as well as a multi-category object recognition task.