Attention-based Neural Bag-of-Features Learning for Sequence Data

TL;DR

This paper introduces a novel 2D-Attention mechanism integrated into Neural Bag-of-Features models, enhancing their ability to focus on relevant information in sequence data and improving performance across various domains.

Contribution

It proposes a flexible attention module called 2D-Attention that can be embedded into NBoF models, providing new interpretability and robustness to noise.

Findings

Improved accuracy in financial forecasting, audio analysis, and medical diagnosis.

Enhanced robustness of NBoF models to noisy data.

Versatile plug-in layer adaptable to different NBoF architectures.

Abstract

In this paper, we propose 2D-Attention (2DA), a generic attention formulation for sequence data, which acts as a complementary computation block that can detect and focus on relevant sources of information for the given learning objective. The proposed attention module is incorporated into the recently proposed Neural Bag of Feature (NBoF) model to enhance its learning capacity. Since 2DA acts as a plug-in layer, injecting it into different computation stages of the NBoF model results in different 2DA-NBoF architectures, each of which possesses a unique interpretation. We conducted extensive experiments in financial forecasting, audio analysis as well as medical diagnosis problems to benchmark the proposed formulations in comparison with existing methods, including the widely used Gated Recurrent Units. Our empirical analysis shows that the proposed attention formulations can not only improve performances of NBoF models but also make them resilient to noisy data.