Learning Subspace-Preserving Sparse Attention Graphs from Heterogeneous Multiview Data

TL;DR

This paper introduces SAGL, a novel method for learning subspace-preserving sparse attention graphs from heterogeneous multiview data, improving unsupervised transfer learning by capturing intrinsic structures.

Contribution

The paper proposes a bilinear attention scheme with dynamic sparsity gating and structured sparse projection to effectively learn subspace-preserving graphs from multiview data.

Findings

SAGL outperforms state-of-the-art methods on benchmark datasets.

The bilinear attention factorization captures asymmetric similarities effectively.

Structured sparse projection ensures subspace preservation in learned graphs.

Abstract

The high-dimensional features extracted from large-scale unlabeled data via various pretrained models with diverse architectures are referred to as heterogeneous multiview data. Most existing unsupervised transfer learning methods fail to faithfully recover intrinsic subspace structures when exploiting complementary information across multiple views. Therefore, a fundamental challenge involves constructing sparse similarity graphs that preserve these underlying subspace structures for achieving semantic alignment across heterogeneous views. In this paper, we propose a sparse attention graph learning (SAGL) method that learns subspace-preserving sparse attention graphs from heterogeneous multiview data. Specifically, we introduce a bilinear attention factorization scheme to capture asymmetric similarities among the high-dimensional features, which breaks the symmetry bottleneck that is inherent in the traditional representation learning techniques. A dynamic sparsity gating mechanism then predicts a feature-specific compression factor for adaptively controlling the topological contributions of neighbors. Furthermore, we employ a structured sparse projection via $\alpha$-entmax to generate subspace-preserving sparse attention graphs for individual views. SAGL leverages these view-specific graphs to conduct sparse information aggregation, yielding discriminative representations for multiview learning tasks. In addition, we provide a rigorous theoretical analysis that bridges differentiable sparse attention and probability simplex constraints. Extensive experiments conducted on multiple benchmark datasets demonstrate that SAGL consistently outperforms the state-of-the-art unsupervised transfer learning approaches.