GILBO: One Metric to Measure Them All

TL;DR

GILBO introduces a simple lower bound to measure the mutual information in latent variable generative models, providing a data-independent complexity metric applicable to VAEs and GANs, with empirical evaluation on multiple datasets.

Contribution

The paper presents GILBO, a novel tractable lower bound on mutual information for latent generative models, enabling complexity assessment independent of data.

Findings

GILBO effectively measures model complexity across different architectures.

Empirical analysis on 1600 models reveals insights into generative model behaviors.

GILBO is applicable to both VAEs and GANs, facilitating comparative studies.

Abstract

We propose a simple, tractable lower bound on the mutual information contained in the joint generative density of any latent variable generative model: the GILBO (Generative Information Lower BOund). It offers a data-independent measure of the complexity of the learned latent variable description, giving the log of the effective description length. It is well-defined for both VAEs and GANs. We compute the GILBO for 800 GANs and VAEs each trained on four datasets (MNIST, FashionMNIST, CIFAR-10 and CelebA) and discuss the results.