Learning Purified Feature Representations from Task-irrelevant Labels

TL;DR

This paper introduces PurifiedLearning, a novel framework that leverages task-irrelevant labels to purify feature representations, improving model generalization on small datasets through theoretical analysis and extensive experiments.

Contribution

It proposes a model-agnostic framework that exploits task-irrelevant features to enhance deep neural network training on limited data, with solid theoretical backing.

Findings

PurifiedLearning improves classification performance on small datasets.

The framework is compatible with any deep neural network architecture.

Experimental results validate the effectiveness of the proposed method.

Abstract

Learning an empirically effective model with generalization using limited data is a challenging task for deep neural networks. In this paper, we propose a novel learning framework called PurifiedLearning to exploit task-irrelevant features extracted from task-irrelevant labels when training models on small-scale datasets. Particularly, we purify feature representations by using the expression of task-irrelevant information, thus facilitating the learning process of classification. Our work is built on solid theoretical analysis and extensive experiments, which demonstrate the effectiveness of PurifiedLearning. According to the theory we proved, PurifiedLearning is model-agnostic and doesn't have any restrictions on the model needed, so it can be combined with any existing deep neural networks with ease to achieve better performance. The source code of this paper will be available in the future for reproducibility.