REVE: Regularizing Deep Learning with Variational Entropy Bound

TL;DR

REVE is a novel regularization method for deep learning that directly targets the class-conditioned entropy of the prediction variable using a variational upper bound, improving generalization.

Contribution

It introduces a new regularization scheme that compresses the class-conditioned entropy of the prediction variable via a variational bound, enhancing neural network training.

Findings

Effective across different neural networks and datasets.

Improves generalization by focusing on the prediction variable.

Provides a tractable loss integrated into training.

Abstract

Studies on generalization performance of machine learning algorithms under the scope of information theory suggest that compressed representations can guarantee good generalization, inspiring many compression-based regularization methods. In this paper, we introduce REVE, a new regularization scheme. Noting that compressing the representation can be sub-optimal, our first contribution is to identify a variable that is directly responsible for the final prediction. Our method aims at compressing the class conditioned entropy of this latter variable. Second, we introduce a variational upper bound on this conditional entropy term. Finally, we propose a scheme to instantiate a tractable loss that is integrated within the training procedure of the neural network and demonstrate its efficiency on different neural networks and datasets.