Learning Sequence Representations by Non-local Recurrent Neural Memory

TL;DR

This paper introduces Non-local Recurrent Neural Memory (NRNM), a novel sequence learning model that captures high-order, long-range dependencies using self-attention and gated recurrent mechanisms, outperforming existing methods across various sequence tasks.

Contribution

The paper proposes NRNM, a new model that effectively models high-order and long-range dependencies in sequences through non-local operations and global interactions.

Findings

NRNM outperforms state-of-the-art methods in sequence classification.

NRNM effectively captures long-range dependencies.

NRNM generalizes well across different sequence modalities.

Abstract

The key challenge of sequence representation learning is to capture the long-range temporal dependencies. Typical methods for supervised sequence representation learning are built upon recurrent neural networks to capture temporal dependencies. One potential limitation of these methods is that they only model one-order information interactions explicitly between adjacent time steps in a sequence, hence the high-order interactions between nonadjacent time steps are not fully exploited. It greatly limits the capability of modeling the long-range temporal dependencies since the temporal features learned by one-order interactions cannot be maintained for a long term due to temporal information dilution and gradient vanishing. To tackle this limitation, we propose the Non-local Recurrent Neural Memory (NRNM) for supervised sequence representation learning, which performs non-local operations \MR{by means of self-attention mechanism} to learn full-order interactions within a sliding temporal memory block and models global interactions between memory blocks in a gated recurrent manner. Consequently, our model is able to capture long-range dependencies. Besides, the latent high-level features contained in high-order interactions can be distilled by our model. We validate the effectiveness and generalization of our NRNM on three types of sequence applications across different modalities, including sequence classification, step-wise sequential prediction and sequence similarity learning. Our model compares favorably against other state-of-the-art methods specifically designed for each of these sequence applications.