SAHDL: Sparse Attention Hypergraph Regularized Dictionary Learning

TL;DR

This paper introduces a novel sparse attention hypergraph regularized dictionary learning method that independently updates attention weights using hypergraph structures, applicable to traditional machine learning, and demonstrates improved performance on benchmark datasets.

Contribution

It proposes a hypergraph based sparse attention mechanism integrated into dictionary learning, independent of deep networks, to capture high-order relationships and local structures.

Findings

Effective in capturing high-order sample relationships

Preserves local structure via hypergraph Laplacian

Shows improved results on benchmark datasets

Abstract

In recent years, the attention mechanism contributes significantly to hypergraph based neural networks. However, these methods update the attention weights with the network propagating. That is to say, this type of attention mechanism is only suitable for deep learning-based methods while not applicable to the traditional machine learning approaches. In this paper, we propose a hypergraph based sparse attention mechanism to tackle this issue and embed it into dictionary learning. More specifically, we first construct a sparse attention hypergraph, asset attention weights to samples by employing the $\ell_1$-norm sparse regularization to mine the high-order relationship among sample features. Then, we introduce the hypergraph Laplacian operator to preserve the local structure for subspace transformation in dictionary learning. Besides, we incorporate the discriminative information into the hypergraph as the guidance to aggregate samples. Unlike previous works, our method updates attention weights independently, does not rely on the deep network. We demonstrate the efficacy of our approach on four benchmark datasets.