Diverse Dictionary Learning

TL;DR

This paper introduces diverse dictionary learning, showing that certain set-theoretic structures of latent variables are identifiable under minimal assumptions, enabling more reliable understanding of hidden data.

Contribution

It formalizes the concept of diverse dictionary learning, providing set-theoretic identifiability results and a simple inductive bias applicable to various models.

Findings

Set operations on latent variables are identifiable with minimal assumptions.

Structural diversity enables full identifiability of all latent variables.

The proposed bias improves latent variable recovery on synthetic and real data.

Abstract

Given only observational data $X = g(Z)$, where both the latent variables $Z$ and the generating process $g$ are unknown, recovering $Z$ is ill-posed without additional assumptions. Existing methods often assume linearity or rely on auxiliary supervision and functional constraints. However, such assumptions are rarely verifiable in practice, and most theoretical guarantees break down under even mild violations, leaving uncertainty about how to reliably understand the hidden world. To make identifiability actionable in the real-world scenarios, we take a complementary view: in the general settings where full identifiability is unattainable, what can still be recovered with guarantees, and what biases could be universally adopted? We introduce the problem of diverse dictionary learning to formalize this view. Specifically, we show that intersections, complements, and symmetric differences of latent variables linked to arbitrary observations, along with the latent-to-observed dependency structure, are still identifiable up to appropriate indeterminacies even without strong assumptions. These set-theoretic results can be composed using set algebra to construct structured and essential views of the hidden world, such as genus-differentia definitions. When sufficient structural diversity is present, they further imply full identifiability of all latent variables. Notably, all identifiability benefits follow from a simple inductive bias during estimation that can be readily integrated into most models. We validate the theory and demonstrate the benefits of the bias on both synthetic and real-world data.