FastTrees: Parallel Latent Tree-Induction for Faster Sequence Encoding

TL;DR

FASTTREES introduces a parallel, non-autoregressive neural module for faster sequence encoding that induces latent tree structures, outperforming existing models on various NLP tasks and enhancing Transformer performance.

Contribution

It presents FASTTREES, a novel parallel tree induction method that improves sequence encoding speed and performance, and can be integrated into Transformer models for better results.

Findings

Achieves competitive or superior performance to ON-LSTM on four sequence tasks.

Enhances Transformer models, improving performance on three sequence transduction tasks.

Outperforms state-of-the-art models on logical inference (+4%) and mathematical language understanding (+8%).

Abstract

Inducing latent tree structures from sequential data is an emerging trend in the NLP research landscape today, largely popularized by recent methods such as Gumbel LSTM and Ordered Neurons (ON-LSTM). This paper proposes FASTTREES, a new general purpose neural module for fast sequence encoding. Unlike most previous works that consider recurrence to be necessary for tree induction, our work explores the notion of parallel tree induction, i.e., imbuing our model with hierarchical inductive biases in a parallelizable, non-autoregressive fashion. To this end, our proposed FASTTREES achieves competitive or superior performance to ON-LSTM on four well-established sequence modeling tasks, i.e., language modeling, logical inference, sentiment analysis and natural language inference. Moreover, we show that the FASTTREES module can be applied to enhance Transformer models, achieving performance gains on three sequence transduction tasks (machine translation, subject-verb agreement and mathematical language understanding), paving the way for modular tree induction modules. Overall, we outperform existing state-of-the-art models on logical inference tasks by +4% and mathematical language understanding by +8%.