Compositional Structures in Neural Embedding and Interaction Decompositions

TL;DR

This paper explores the relationship between linear algebraic structures in neural embeddings and probabilistic independence, providing a formal framework for understanding emergent compositional patterns in data representations.

Contribution

It introduces a formal characterization of compositional structures via interaction decompositions and establishes conditions for their presence in neural network representations.

Findings

Link between linear algebraic structures and probabilistic independence

Necessary and sufficient conditions for compositional structures

Formal framework for analyzing data representation patterns

Abstract

We describe a basic correspondence between linear algebraic structures within vector embeddings in artificial neural networks and conditional independence constraints on the probability distributions modeled by these networks. Our framework aims to shed light on the emergence of structural patterns in data representations, a phenomenon widely acknowledged but arguably still lacking a solid formal grounding. Specifically, we introduce a characterization of compositional structures in terms of "interaction decompositions," and we establish necessary and sufficient conditions for the presence of such structures within the representations of a model.