Learning Structured Latent Factors from Dependent Data:A Generative Model Framework from Information-Theoretic Perspective

TL;DR

This paper introduces a novel generative modeling framework that learns structured latent factors from dependent data, leveraging information-theoretic principles to automatically infer and enforce various dependency structures.

Contribution

It proposes a unified, principled approach to learn semantically meaningful latent factors with desired structures, extending multivariate information bottleneck theory.

Findings

Framework can automatically estimate dependency structures from data.

Unifies various existing generative models under a common framework.

Applicable to multi-modal data, fairness, and invariant risk minimization.

Abstract

Learning controllable and generalizable representation of multivariate data with desired structural properties remains a fundamental problem in machine learning. In this paper, we present a novel framework for learning generative models with various underlying structures in the latent space. We represent the inductive bias in the form of mask variables to model the dependency structure in the graphical model and extend the theory of multivariate information bottleneck to enforce it. Our model provides a principled approach to learn a set of semantically meaningful latent factors that reflect various types of desired structures like capturing correlation or encoding invariance, while also offering the flexibility to automatically estimate the dependency structure from data. We show that our framework unifies many existing generative models and can be applied to a variety of tasks including multi-modal data modeling, algorithmic fairness, and invariant risk minimization.