The emergent algebraic structure of RNNs and embeddings in NLP

TL;DR

This paper explores the algebraic structure of RNNs and embeddings in NLP, revealing that words embed in Lie groups and RNNs form nonlinear group representations, leading to new neural network architectures.

Contribution

It uncovers the Lie group structure in word embeddings and RNNs, and proposes novel recurrent neural network models based on these algebraic insights.

Findings

Words embed in Lie groups

RNNs form nonlinear group representations

Proposed new recurrent neural network architectures

Abstract

We examine the algebraic and geometric properties of a uni-directional GRU and word embeddings trained end-to-end on a text classification task. A hyperparameter search over word embedding dimension, GRU hidden dimension, and a linear combination of the GRU outputs is performed. We conclude that words naturally embed themselves in a Lie group and that RNNs form a nonlinear representation of the group. Appealing to these results, we propose a novel class of recurrent-like neural networks and a word embedding scheme.