Learning Discrete Concepts in Latent Hierarchical Models

TL;DR

This paper formalizes the learning of discrete, hierarchical concepts as latent causal variables in high-dimensional data, providing conditions for their identification and demonstrating implications for models like diffusion models.

Contribution

It introduces a novel hierarchical causal model for concepts in high-dimensional data and establishes conditions for their unsupervised learning, extending prior work to complex structures and continuous variables.

Findings

Theoretical conditions for identifying hierarchical causal concepts.

Empirical validation with synthetic data.

Implications for understanding latent diffusion models.

Abstract

Learning concepts from natural high-dimensional data (e.g., images) holds potential in building human-aligned and interpretable machine learning models. Despite its encouraging prospect, formalization and theoretical insights into this crucial task are still lacking. In this work, we formalize concepts as discrete latent causal variables that are related via a hierarchical causal model that encodes different abstraction levels of concepts embedded in high-dimensional data (e.g., a dog breed and its eye shapes in natural images). We formulate conditions to facilitate the identification of the proposed causal model, which reveals when learning such concepts from unsupervised data is possible. Our conditions permit complex causal hierarchical structures beyond latent trees and multi-level directed acyclic graphs in prior work and can handle high-dimensional, continuous observed variables, which is well-suited for unstructured data modalities such as images. We substantiate our theoretical claims with synthetic data experiments. Further, we discuss our theory's implications for understanding the underlying mechanisms of latent diffusion models and provide corresponding empirical evidence for our theoretical insights.