Compact Compositional Models

TL;DR

This paper introduces a new composition rule and initialization method for learning compact, interpretable models of binary data, emphasizing diverse expert interactions and improved training procedures.

Contribution

It proposes a novel composition rule that discourages similar expert focus and a sequential initialization process for better model learning.

Findings

Models learned are more intuitive and interpretable

The approach improves the diversity of expert focus

Experimental results demonstrate effective learning of low-dimensional representations

Abstract

Learning compact and interpretable representations is a very natural task, which has not been solved satisfactorily even for simple binary datasets. In this paper, we review various ways of composing experts for binary data and argue that competitive forms of interaction are best suited to learn low-dimensional representations. We propose a new composition rule that discourages experts from focusing on similar structures and that penalizes opposing votes strongly so that abstaining from voting becomes more attractive. We also introduce a novel sequential initialization procedure, which is based on a process of oversimplification and correction. Experiments show that with our approach very intuitive models can be learned.