Self-Reinforcement Attention Mechanism For Tabular Learning

TL;DR

This paper introduces Self-Reinforcement Attention (SRA), a novel attention mechanism for tabular data that enhances interpretability and performance, especially on imbalanced datasets, by providing feature relevance weights.

Contribution

The paper proposes SRA, an innovative attention mechanism that generates feature relevance weights to improve interpretability and effectiveness in tabular learning models.

Findings

SRA improves model interpretability by providing feature relevance weights.

SRA enhances performance on synthetic and real-world imbalanced datasets.

SRA can be integrated with existing baseline models effectively.

Abstract

Apart from the high accuracy of machine learning models, what interests many researchers in real-life problems (e.g., fraud detection, credit scoring) is to find hidden patterns in data; particularly when dealing with their challenging imbalanced characteristics. Interpretability is also a key requirement that needs to accompany the used machine learning model. In this concern, often, intrinsically interpretable models are preferred to complex ones, which are in most cases black-box models. Also, linear models are used in some high-risk fields to handle tabular data, even if performance must be sacrificed. In this paper, we introduce Self-Reinforcement Attention (SRA), a novel attention mechanism that provides a relevance of features as a weight vector which is used to learn an intelligible representation. This weight is then used to reinforce or reduce some components of the raw input through element-wise vector multiplication. Our results on synthetic and real-world imbalanced data show that our proposed SRA block is effective in end-to-end combination with baseline models.