Towards Further Understanding of Sparse Filtering via Information Bottleneck

TL;DR

This paper investigates the behavior of sparse filtering through an information-theoretic lens, using the information bottleneck framework to analyze its dynamics and support the conjecture that FDL algorithms optimize mutual information and entropy.

Contribution

It provides empirical evidence supporting the conjecture that sparse filtering's behavior can be characterized by information-theoretic quantities, extending IB analysis to FDL algorithms.

Findings

Information planes reveal the dynamics of sparse filtering.

Empirical support for the mutual information and entropy optimization conjecture.

Insights may guide development of new information-theoretic assessment tools.

Abstract

In this paper we examine a formalization of feature distribution learning (FDL) in information-theoretic terms relying on the analytical approach and on the tools already used in the study of the information bottleneck (IB). It has been conjectured that the behavior of FDL algorithms could be expressed as an optimization problem over two information-theoretic quantities: the mutual information of the data with the learned representations and the entropy of the learned distribution. In particular, such a formulation was offered in order to explain the success of the most prominent FDL algorithm, sparse filtering (SF). This conjecture was, however, left unproven. In this work, we aim at providing preliminary empirical support to this conjecture by performing experiments reminiscent of the work done on deep neural networks in the context of the IB research. Specifically, we borrow the idea of using information planes to analyze the behavior of the SF algorithm and gain insights on its dynamics. A confirmation of the conjecture about the dynamics of FDL may provide solid ground to develop information-theoretic tools to assess the quality of the learning process in FDL, and it may be extended to other unsupervised learning algorithms.