Tree-structured Attention with Hierarchical Accumulation

TL;DR

This paper introduces a novel hierarchical accumulation method that encodes parse tree structures into self-attention efficiently, outperforming existing models in translation and classification tasks, and effectively leveraging hierarchical priors.

Contribution

The paper proposes a new hierarchical accumulation technique that encodes tree structures into self-attention with constant time complexity, bridging the gap between Transformers and Tree-LSTMs.

Findings

Outperforms SOTA in four IWSLT translation tasks and WMT'14 English-German translation.

Improves performance on three text classification tasks over Transformer and Tree-LSTM.

Using hierarchical priors helps mitigate data scarcity and favors phrase-level attention.

Abstract

Incorporating hierarchical structures like constituency trees has been shown to be effective for various natural language processing (NLP) tasks. However, it is evident that state-of-the-art (SOTA) sequence-based models like the Transformer struggle to encode such structures inherently. On the other hand, dedicated models like the Tree-LSTM, while explicitly modeling hierarchical structures, do not perform as efficiently as the Transformer. In this paper, we attempt to bridge this gap with "Hierarchical Accumulation" to encode parse tree structures into self-attention at constant time complexity. Our approach outperforms SOTA methods in four IWSLT translation tasks and the WMT'14 English-German translation task. It also yields improvements over Transformer and Tree-LSTM on three text classification tasks. We further demonstrate that using hierarchical priors can compensate for data shortage, and that our model prefers phrase-level attentions over token-level attentions.