Deep Representation Learning with an Information-theoretic Loss

TL;DR

This paper introduces an information-theoretic loss for deep representation learning that enhances class separation and compactness, improving anomaly and out-of-distribution detection by better structuring the embedded space.

Contribution

It develops a novel loss based on the Information Bottleneck principle to improve class separation and compactness in deep embeddings, extending deep data description models.

Findings

Improves segmentation of normal classes in deep feature space.

Enhances detection of out-of-distribution samples.

Extends existing deep data description models.

Abstract

This paper proposes a deep representation learning using an information-theoretic loss with an aim to increase the inter-class distances as well as within-class similarity in the embedded space. Tasks such as anomaly and out-of-distribution detection, in which test samples comes from classes unseen in training, are problematic for deep neural networks. For such tasks, it is not sufficient to merely discriminate between known classes. Our intuition is to represent the known classes in compact and separated embedded regions in order to decrease the possibility of known and unseen classes overlapping in the embedded space. We derive a loss from Information Bottleneck principle, which reflects the inter-class distances as well as the compactness within classes, thus will extend the existing deep data description models. Our empirical study shows that the proposed model improves the segmentation of normal classes in the deep feature space, and subsequently contributes to identifying out-of-distribution samples.