Learning and Compositionality: a Unification Attempt via Connectionist Probabilistic Programming

TL;DR

This paper proposes a unified framework called Connectionist Probabilistic Programs (CPPs) that aims to integrate learning and compositionality, key mechanisms for human-like intelligence, by combining neural structures with probabilistic program semantics.

Contribution

It introduces CPPs as a novel unification framework, analyzing neural and symbolic methods, and demonstrates initial sequence modeling results using Bayesian inference.

Findings

Successful concept and relation extraction from raw data

Framework demonstrates potential for compositional learning

Initial sequence modeling results show promise

Abstract

We consider learning and compositionality as the key mechanisms towards simulating human-like intelligence. While each mechanism is successfully achieved by neural networks and symbolic AIs, respectively, it is the combination of the two mechanisms that makes human-like intelligence possible. Despite the numerous attempts on building hybrid neuralsymbolic systems, we argue that our true goal should be unifying learning and compositionality, the core mechanisms, instead of neural and symbolic methods, the surface approaches to achieve them. In this work, we review and analyze the strengths and weaknesses of neural and symbolic methods by separating their forms and meanings (structures and semantics), and propose Connectionist Probabilistic Program (CPPs), a framework that connects connectionist structures (for learning) and probabilistic program semantics (for compositionality). Under the framework, we design a CPP extension for small scale sequence modeling and provide a learning algorithm based on Bayesian inference. Although challenges exist in learning complex patterns without supervision, our early results demonstrate CPP's successful extraction of concepts and relations from raw sequential data, an initial step towards compositional learning.