Learning a Deep Generative Model like a Program: the Free Category Prior

TL;DR

This paper introduces the free category prior, a novel formalism that enables neural networks to learn structured, compositional probabilistic programs end-to-end, challenging traditional symbolic assumptions in AI.

Contribution

It presents a new formalism for neural networks to learn program structure and parameters simultaneously, emphasizing compositionality via network structure rather than symbolic representations.

Findings

Neural networks can represent compositional programs using the free category prior.

The approach allows end-to-end learning of program structure and parameters.

It provides a counterexample to the assumption that brain-like computation requires symbolic representations.

Abstract

Humans surpass the cognitive abilities of most other animals in our ability to "chunk" concepts into words, and then combine the words to combine the concepts. In this process, we make "infinite use of finite means", enabling us to learn new concepts quickly and nest concepts within each-other. While program induction and synthesis remain at the heart of foundational theories of artificial intelligence, only recently has the community moved forward in attempting to use program learning as a benchmark task itself. The cognitive science community has thus often assumed that if the brain has simulation and reasoning capabilities equivalent to a universal computer, then it must employ a serialized, symbolic representation. Here we confront that assumption, and provide a counterexample in which compositionality is expressed via network structure: the free category prior over programs. We show how our formalism allows neural networks to serve as primitives in probabilistic programs. We learn both program structure and model parameters end-to-end.