A Compositional Approach to Language Modeling

TL;DR

This paper introduces a novel language model that marginalizes over all possible composition trees, removing structural assumptions, and demonstrates significant improvements over existing RNN-based models using a new evaluation metric.

Contribution

It proposes a compositional language modeling approach that eliminates structural assumptions and achieves substantial performance gains over traditional models.

Findings

Over 100% improvement in Contrastive Entropy over RNN models

Models marginalize over all possible composition trees

Demonstrates the limitations of the linear chain assumption

Abstract

Traditional language models treat language as a finite state automaton on a probability space over words. This is a very strong assumption when modeling something inherently complex such as language. In this paper, we challenge this by showing how the linear chain assumption inherent in previous work can be translated into a sequential composition tree. We then propose a new model that marginalizes over all possible composition trees thereby removing any underlying structural assumptions. As the partition function of this new model is intractable, we use a recently proposed sentence level evaluation metric Contrastive Entropy to evaluate our model. Given this new evaluation metric, we report more than 100% improvement across distortion levels over current state of the art recurrent neural network based language models.