Representation Learning via Variational Bayesian Networks

TL;DR

The paper introduces Variational Bayesian Network (VBN), a new model for entity representation that leverages hierarchical and relational information, modeling uncertainty to improve performance on scarce data, especially for long-tail entities.

Contribution

VBN is a novel Bayesian model that uses hierarchical priors and explicit relations to enhance entity representations, particularly in data-scarce scenarios.

Findings

VBN outperforms existing methods on linguistic, recommendation, and medical tasks.

VBN effectively models uncertainty and long-tail entities.

VBN provides scalable Bayesian inference with a new optimization algorithm.

Abstract

We present Variational Bayesian Network (VBN) - a novel Bayesian entity representation learning model that utilizes hierarchical and relational side information and is particularly useful for modeling entities in the ``long-tail'', where the data is scarce. VBN provides better modeling for long-tail entities via two complementary mechanisms: First, VBN employs informative hierarchical priors that enable information propagation between entities sharing common ancestors. Additionally, VBN models explicit relations between entities that enforce complementary structure and consistency, guiding the learned representations towards a more meaningful arrangement in space. Second, VBN represents entities by densities (rather than vectors), hence modeling uncertainty that plays a complementary role in coping with data scarcity. Finally, we propose a scalable Variational Bayes optimization algorithm that enables fast approximate Bayesian inference. We evaluate the effectiveness of VBN on linguistic, recommendations, and medical inference tasks. Our findings show that VBN outperforms other existing methods across multiple datasets, and especially in the long-tail.