Utilizing Class Information for Deep Network Representation Shaping

TL;DR

This paper introduces class-wise regularizers for deep network representations that improve classification performance by explicitly shaping features per class, demonstrating significant gains on benchmark datasets.

Contribution

The paper proposes two novel class-wise regularizers, cw-CR and cw-VR, which explicitly utilize class information to enhance feature independence and compactness, outperforming traditional regularizers.

Findings

Significant performance improvements over baseline and L1/L2 regularization on 21 out of 22 tasks.

cw-VR achieved the best results on 13 tasks, including ResNet-32/110.

Regularizers are simple, efficient, and effectively shape representations as visualized.

Abstract

Statistical characteristics of deep network representations, such as sparsity and correlation, are known to be relevant to the performance and interpretability of deep learning. When a statistical characteristic is desired, often an adequate regularizer can be designed and applied during the training phase. Typically, such a regularizer aims to manipulate a statistical characteristic over all classes together. For classification tasks, however, it might be advantageous to enforce the desired characteristic per class such that different classes can be better distinguished. Motivated by the idea, we design two class-wise regularizers that explicitly utilize class information: class-wise Covariance Regularizer (cw-CR) and class-wise Variance Regularizer (cw-VR). cw-CR targets to reduce the covariance of representations calculated from the same class samples for encouraging feature independence. cw-VR is similar, but variance instead of covariance is targeted to improve feature compactness. For the sake of completeness, their counterparts without using class information, Covariance Regularizer (CR) and Variance Regularizer (VR), are considered together. The four regularizers are conceptually simple and computationally very efficient, and the visualization shows that the regularizers indeed perform distinct representation shaping. In terms of classification performance, significant improvements over the baseline and L1/L2 weight regularization methods were found for 21 out of 22 tasks over popular benchmark datasets. In particular, cw-VR achieved the best performance for 13 tasks including ResNet-32/110.