Discrete, compositional, and symbolic representations through attractor dynamics

TL;DR

This paper introduces a neural stochastic dynamical systems model that unifies symbolic and sub-symbolic processing via attractor dynamics, enabling unsupervised learning of discrete, compositional representations akin to human cognition.

Contribution

The work presents a novel model that derives symbolic representations from continuous neural dynamics without pre-defined primitives, bridging the gap between symbolic and sub-symbolic AI.

Findings

Segments representational space into discrete basins

Learns to sample diverse symbolic attractor states

Reflects semanticity and compositionality in representations

Abstract

Symbolic systems are powerful frameworks for modeling cognitive processes as they encapsulate the rules and relationships fundamental to many aspects of human reasoning and behavior. Central to these models are systematicity, compositionality, and productivity, making them invaluable in both cognitive science and artificial intelligence. However, certain limitations remain. For instance, the integration of structured symbolic processes and latent sub-symbolic processes has been implemented at the computational level through fiat methods such as quantization or softmax sampling, which assume, rather than derive, the operations underpinning discretization and symbolicization. In this work, we introduce a novel neural stochastic dynamical systems model that integrates attractor dynamics with symbolic representations to model cognitive processes akin to the probabilistic language of thought (PLoT). Our model segments the continuous representational space into discrete basins, with attractor states corresponding to symbolic sequences, that reflect the semanticity and compositionality characteristic of symbolic systems through unsupervised learning, rather than relying on pre-defined primitives. Moreover, like PLoT, our model learns to sample a diverse distribution of attractor states that reflect the mutual information between the input data and the symbolic encodings. This approach establishes a unified framework that integrates both symbolic and sub-symbolic processing through neural dynamics, a neuro-plausible substrate with proven expressivity in AI, offering a more comprehensive model that mirrors the complex duality of cognitive operations.