Learning by Analogy: A Causal Framework for Composition Generalization

TL;DR

This paper introduces a causal, hierarchical framework for compositional generalization that models how high-level concepts can be decomposed and recombined, supported by theoretical proofs and improved benchmark results.

Contribution

It formalizes a causal, hierarchical approach to compositional generalization, proving the recoverability of the latent structure from observable data, and demonstrates empirical improvements.

Findings

Hierarchical data-generating process supports complex concept relations

Latent structure is identifiable from data like text-image pairs

Significant performance improvements on benchmark datasets

Abstract

Compositional generalization -- the ability to understand and generate novel combinations of learned concepts -- enables models to extend their capabilities beyond limited experiences. While effective, the data structures and principles that enable this crucial capability remain poorly understood. We propose that compositional generalization fundamentally requires decomposing high-level concepts into basic, low-level concepts that can be recombined across similar contexts, similar to how humans draw analogies between concepts. For example, someone who has never seen a peacock eating rice can envision this scene by relating it to their previous observations of a chicken eating rice. In this work, we formalize these intuitive processes using principles of causal modularity and minimal changes. We introduce a hierarchical data-generating process that naturally encodes different levels of concepts and their interaction mechanisms. Theoretically, we demonstrate that this approach enables compositional generalization supporting complex relations between composed concepts, advancing beyond prior work that assumes simpler interactions like additive effects. Critically, we also prove that this latent hierarchical structure is provably recoverable (identifiable) from observable data like text-image pairs, a necessary step for learning such a generative process. To validate our theory, we apply insights from our theoretical framework and achieve significant improvements on benchmark datasets.