Long Short-Term Attention

TL;DR

This paper introduces Long Short-Term Attention (LSTA), a new model that integrates attention mechanisms into LSTM cells, enhancing sequence learning by focusing on important information and outperforming traditional LSTM models.

Contribution

The paper presents LSTA, a novel model that combines attention with LSTM cells, enabling better focus on sequence information and improved performance.

Findings

LSTA outperforms LSTM on sequence learning tasks.

LSTA effectively focuses on important sequence information.

Experiments demonstrate superior performance of LSTA.

Abstract

Attention is an important cognition process of humans, which helps humans concentrate on critical information during their perception and learning. However, although many machine learning models can remember information of data, they have no the attention mechanism. For example, the long short-term memory (LSTM) network is able to remember sequential information, but it cannot pay special attention to part of the sequences. In this paper, we present a novel model called long short-term attention (LSTA), which seamlessly integrates the attention mechanism into the inner cell of LSTM. More than processing long short term dependencies, LSTA can focus on important information of the sequences with the attention mechanism. Extensive experiments demonstrate that LSTA outperforms LSTM and related models on the sequence learning tasks.