Neural Semantic Encoders

TL;DR

Neural Semantic Encoders (NSE) are memory-augmented neural networks designed for natural language understanding, capable of evolving memory and handling multiple shared memories, achieving state-of-the-art results across various NLP tasks.

Contribution

Introduction of NSE with a novel memory update rule and flexible memory management, demonstrating effectiveness across multiple NLP applications.

Findings

Achieved state-of-the-art performance on five NLP tasks.

Improved neural machine translation BLEU score by approximately 1.0.

Demonstrated the flexibility of shared and multiple memories.

Abstract

We present a memory augmented neural network for natural language understanding: Neural Semantic Encoders. NSE is equipped with a novel memory update rule and has a variable sized encoding memory that evolves over time and maintains the understanding of input sequences through read}, compose and write operations. NSE can also access multiple and shared memories. In this paper, we demonstrated the effectiveness and the flexibility of NSE on five different natural language tasks: natural language inference, question answering, sentence classification, document sentiment analysis and machine translation where NSE achieved state-of-the-art performance when evaluated on publically available benchmarks. For example, our shared-memory model showed an encouraging result on neural machine translation, improving an attention-based baseline by approximately 1.0 BLEU.