Competing Mutual Information Constraints with Stochastic Competition-based Activations for Learning Diversified Representations

TL;DR

This paper introduces a novel approach combining stochastic competition-based activations with information-theoretic principles to learn diversified, sparse representations in neural networks, demonstrated through image classification benchmarks.

Contribution

It proposes replacing traditional non-linear activations with stochastic local winner-takes-all units and integrates information-theoretic constraints within a Bayesian framework for improved representation diversity.

Findings

Networks achieve significant discriminative capabilities

Method enables inference of essential network sub-parts

Approach allows principled analysis of intermediate representations

Abstract

This work aims to address the long-established problem of learning diversified representations. To this end, we combine information-theoretic arguments with stochastic competition-based activations, namely Stochastic Local Winner-Takes-All (LWTA) units. In this context, we ditch the conventional deep architectures commonly used in Representation Learning, that rely on non-linear activations; instead, we replace them with sets of locally and stochastically competing linear units. In this setting, each network layer yields sparse outputs, determined by the outcome of the competition between units that are organized into blocks of competitors. We adopt stochastic arguments for the competition mechanism, which perform posterior sampling to determine the winner of each block. We further endow the considered networks with the ability to infer the sub-part of the network that is essential for modeling the data at hand; we impose appropriate stick-breaking priors to this end. To further enrich the information of the emerging representations, we resort to information-theoretic principles, namely the Information Competing Process (ICP). Then, all the components are tied together under the stochastic Variational Bayes framework for inference. We perform a thorough experimental investigation for our approach using benchmark datasets on image classification. As we experimentally show, the resulting networks yield significant discriminative representation learning abilities. In addition, the introduced paradigm allows for a principled investigation mechanism of the emerging intermediate network representations.