Deep Discrete Encoders: Identifiable Deep Generative Models for Rich Data with Discrete Latent Layers

TL;DR

This paper introduces Deep Discrete Encoders (DDEs), a new class of interpretable deep generative models with discrete latent layers that are identifiable, scalable, and applicable to diverse data types, enhancing interpretability and statistical reliability.

Contribution

The paper proposes a novel identifiable deep generative model with discrete latent layers, providing theoretical identifiability conditions and an efficient estimation algorithm, advancing interpretability and statistical understanding.

Findings

Theoretical identifiability conditions for DDEs are established.

The proposed estimation pipeline is scalable and effective for high-dimensional data.

DDEs perform well in real-world applications like topic modeling and image learning.

Abstract

In the era of generative AI, deep generative models (DGMs) with latent representations have gained tremendous popularity. Despite their impressive empirical performance, the statistical properties of these models remain underexplored. DGMs are often overparametrized, non-identifiable, and uninterpretable black boxes, raising serious concerns when deploying them in high-stakes applications. Motivated by this, we propose interpretable deep generative models for rich data types with discrete latent layers, called Deep Discrete Encoders (DDEs). A DDE is a directed graphical model with multiple binary latent layers. Theoretically, we propose transparent identifiability conditions for DDEs, which imply progressively smaller sizes of the latent layers as they go deeper. Identifiability ensures consistent parameter estimation and inspires an interpretable design of the deep architecture. Computationally, we propose a scalable estimation pipeline of a layerwise nonlinear spectral initialization followed by a penalized stochastic approximation EM algorithm. This procedure can efficiently estimate models with exponentially many latent components. Extensive simulation studies for high-dimensional data and deep architectures validate our theoretical results and demonstrate the excellent performance of our algorithms. We apply DDEs to three diverse real datasets with different data types to perform hierarchical topic modeling, image representation learning, and response time modeling in educational testing.