A Deep Network with Visual Text Composition Behavior

TL;DR

This paper introduces a deep neural network that not only performs well on text classification but also demonstrates compositional behavior by hierarchically constructing representations from words to sentences.

Contribution

The paper presents a novel deep network architecture that exhibits compositionality, revealing how hierarchical representations are formed in neural models for text.

Findings

Lower layers focus on individual words with attention weights.

Higher layers compose phrases and clauses progressively.

The network achieves competitive accuracy in text classification.

Abstract

While natural languages are compositional, how state-of-the-art neural models achieve compositionality is still unclear. We propose a deep network, which not only achieves competitive accuracy for text classification, but also exhibits compositional behavior. That is, while creating hierarchical representations of a piece of text, such as a sentence, the lower layers of the network distribute their layer-specific attention weights to individual words. In contrast, the higher layers compose meaningful phrases and clauses, whose lengths increase as the networks get deeper until fully composing the sentence.