Language Modeling Using Tensor Trains

TL;DR

This paper introduces a tensor train-based language model that efficiently captures sentence structures in exponential space, unifying several RNN architectures and outperforming standard RNNs on language tasks.

Contribution

The paper presents the Tensor Train Language Model (TTLM), a novel tensor network approach that unifies and extends existing RNN architectures for improved language modeling.

Findings

TTLM variants outperform vanilla RNNs on language tasks

TTLM unifies second-order RNNs, RACs, and Multiplicative Integration RNNs

Efficient low-dimensional computation of sentence probabilities

Abstract

We propose a novel tensor network language model based on the simplest tensor network (i.e., tensor trains), called `Tensor Train Language Model' (TTLM). TTLM represents sentences in an exponential space constructed by the tensor product of words, but computing the probabilities of sentences in a low-dimensional fashion. We demonstrate that the architectures of Second-order RNNs, Recurrent Arithmetic Circuits (RACs), and Multiplicative Integration RNNs are, essentially, special cases of TTLM. Experimental evaluations on real language modeling tasks show that the proposed variants of TTLM (i.e., TTLM-Large and TTLM-Tiny) outperform the vanilla Recurrent Neural Networks (RNNs) with low-scale of hidden units. (The code is available at https://github.com/shuishen112/tensortrainlm.)