Integrating Dependency Tree Into Self-attention for Sentence Representation

TL;DR

This paper introduces Dependency-Transformer, a novel model that integrates dependency tree structures into self-attention mechanisms to improve sentence representation while maintaining parallel processing efficiency.

Contribution

It proposes a relation-attention mechanism that incorporates dependency labels into self-attention, enhancing syntax encoding without sacrificing parallelization.

Findings

Outperforms or matches state-of-the-art on four sentence representation tasks.

Maintains high computational efficiency due to parallelizable architecture.

Effectively encodes dependency and positional relations in sentences.

Abstract

Recent progress on parse tree encoder for sentence representation learning is notable. However, these works mainly encode tree structures recursively, which is not conducive to parallelization. On the other hand, these works rarely take into account the labels of arcs in dependency trees. To address both issues, we propose Dependency-Transformer, which applies a relation-attention mechanism that works in concert with the self-attention mechanism. This mechanism aims to encode the dependency and the spatial positional relations between nodes in the dependency tree of sentences. By a score-based method, we successfully inject the syntax information without affecting Transformer's parallelizability. Our model outperforms or is comparable to the state-of-the-art methods on four tasks for sentence representation and has obvious advantages in computational efficiency.