An Iterative Contextualization Algorithm with Second-Order Attention

TL;DR

This paper introduces an iterative contextualization algorithm utilizing a novel second-order attention mechanism, which improves word representation aggregation for text classification tasks.

Contribution

It presents a new iterative algorithm with second-order attention for better contextual word representations in NLP.

Findings

Achieved strong results on multiple text classification benchmarks.

Demonstrated the effectiveness of second-order attention in iterative context adjustment.

Improved accuracy over baseline models in experiments.

Abstract

Combining the representations of the words that make up a sentence into a cohesive whole is difficult, since it needs to account for the order of words, and to establish how the words present relate to each other. The solution we propose consists in iteratively adjusting the context. Our algorithm starts with a presumably erroneous value of the context, and adjusts this value with respect to the tokens at hand. In order to achieve this, representations of words are built combining their symbolic embedding with a positional encoding into single vectors. The algorithm then iteratively weighs and aggregates these vectors using our novel second-order attention mechanism. Our models report strong results in several well-known text classification tasks.