Self-Adaptive Hierarchical Sentence Model

TL;DR

AdaSent is a self-adaptive hierarchical sentence model that constructs multi-level representations for improved NLP task performance, effectively addressing gradient vanishing issues.

Contribution

It introduces a recursive gated composition mechanism and a competitive gating network for adaptive representation selection in sentence modeling.

Findings

Achieves superior classification accuracy on 5 benchmark datasets.

Automatically selects appropriate hierarchical representations during training.

Mitigates gradient vanishing in recursive models.

Abstract

The ability to accurately model a sentence at varying stages (e.g., word-phrase-sentence) plays a central role in natural language processing. As an effort towards this goal we propose a self-adaptive hierarchical sentence model (AdaSent). AdaSent effectively forms a hierarchy of representations from words to phrases and then to sentences through recursive gated local composition of adjacent segments. We design a competitive mechanism (through gating networks) to allow the representations of the same sentence to be engaged in a particular learning task (e.g., classification), therefore effectively mitigating the gradient vanishing problem persistent in other recursive models. Both qualitative and quantitative analysis shows that AdaSent can automatically form and select the representations suitable for the task at hand during training, yielding superior classification performance over competitor models on 5 benchmark data sets.