Learning to Embed Sentences Using Attentive Recursive Trees

TL;DR

This paper introduces AR-Tree, an attentive recursive tree model that dynamically emphasizes important words in sentence embeddings, improving performance on sentence understanding tasks.

Contribution

The paper proposes a novel attentive recursive tree model with reinforced training that highlights task-informative words in sentence embeddings.

Findings

AR-Tree outperforms state-of-the-art methods on three tasks.

Dynamic word importance improves embedding quality.

Reinforced training enhances model performance.

Abstract

Sentence embedding is an effective feature representation for most deep learning-based NLP tasks. One prevailing line of methods is using recursive latent tree-structured networks to embed sentences with task-specific structures. However, existing models have no explicit mechanism to emphasize task-informative words in the tree structure. To this end, we propose an Attentive Recursive Tree model (AR-Tree), where the words are dynamically located according to their importance in the task. Specifically, we construct the latent tree for a sentence in a proposed important-first strategy, and place more attentive words nearer to the root; thus, AR-Tree can inherently emphasize important words during the bottom-up composition of the sentence embedding. We propose an end-to-end reinforced training strategy for AR-Tree, which is demonstrated to consistently outperform, or be at least comparable to, the state-of-the-art sentence embedding methods on three sentence understanding tasks.