A Hierarchical Approach to Neural Context-Aware Modeling

TL;DR

This paper introduces a hierarchical neural network model that captures multi-level narrative context to improve semantic error detection, significantly outperforming baseline models in accuracy.

Contribution

The paper proposes a novel multi-layered neural architecture that effectively models extended narratives at multiple levels, enhancing language processing tasks.

Findings

Hierarchical model improves error detection accuracy by up to 20%

Model captures context at word, sentence, and narrative levels

Outperforms context-agnostic baselines in experiments

Abstract

We present a new recurrent neural network topology to enhance state-of-the-art machine learning systems by incorporating a broader context. Our approach overcomes recent limitations with extended narratives through a multi-layered computational approach to generate an abstract context representation. Therefore, the developed system captures the narrative on word-level, sentence-level, and context-level. Through the hierarchical set-up, our proposed model summarizes the most salient information on each level and creates an abstract representation of the extended context. We subsequently use this representation to enhance neural language processing systems on the task of semantic error detection. To show the potential of the newly introduced topology, we compare the approach against a context-agnostic set-up including a standard neural language model and a supervised binary classification network. The performance measures on the error detection task show the advantage of the hierarchical context-aware topologies, improving the baseline by 12.75% relative for unsupervised models and 20.37% relative for supervised models.