SAG-VAE: End-to-end Joint Inference of Data Representations and Feature Relations

TL;DR

SAG-VAE introduces an end-to-end model that jointly learns data representations and feature relations using self-attention graph networks, enhancing interpretability, robustness, and generative capabilities in data modeling.

Contribution

The paper presents SAG-VAE, a novel VAE architecture that simultaneously infers feature relations and data representations through a learnable graph structure and self-attention mechanisms.

Findings

SAG-VAE can approximately recover feature relations from data.

The model demonstrates robustness to noise and perturbations.

It effectively generates new data via graph convolution.

Abstract

Variational Autoencoders (VAEs) are powerful in data representation inference, but it cannot learn relations between features with its vanilla form and common variations. The ability to capture relations within data can provide the much needed inductive bias necessary for building more robust Machine Learning algorithms with more interpretable results. In this paper, inspired by recent advances in relational learning using Graph Neural Networks, we propose the Self-Attention Graph Variational AutoEncoder (SAG-VAE) network which can simultaneously learn feature relations and data representations in an end-to-end manner. SAG-VAE is trained by jointly inferring the posterior distribution of two types of latent variables, which denote the data representation and a shared graph structure, respectively. Furthermore, we introduce a novel self-attention graph network that improves the generative capabilities of SAG-VAE by parameterizing the generative distribution allowing SAG-VAE to generate new data via graph convolution, while still trainable via backpropagation. A learnable relational graph representation enhances SAG-VAE's robustness to perturbation and noise, while also providing deeper intuition into model performance. Experiments based on graphs show that SAG-VAE is capable of approximately retrieving edges and links between nodes based entirely on feature observations. Finally, results on image data illustrate that SAG-VAE is fairly robust against perturbations in image reconstruction and sampling.