On the Binding Problem in Artificial Neural Networks

TL;DR

This paper identifies the binding problem as a key obstacle to human-level generalization in neural networks and proposes a framework inspired by neuroscience to develop symbol-like, compositional representations for improved AI generalization.

Contribution

It introduces a unifying framework for dynamic binding in neural networks, combining insights from neuroscience and machine learning to foster symbolic, compositional understanding.

Findings

Survey of mechanisms from neuroscience and machine learning

Identification of inductive biases for symbolic processing

Framework for forming and using meaningful entities

Abstract

Contemporary neural networks still fall short of human-level generalization, which extends far beyond our direct experiences. In this paper, we argue that the underlying cause for this shortcoming is their inability to dynamically and flexibly bind information that is distributed throughout the network. This binding problem affects their capacity to acquire a compositional understanding of the world in terms of symbol-like entities (like objects), which is crucial for generalizing in predictable and systematic ways. To address this issue, we propose a unifying framework that revolves around forming meaningful entities from unstructured sensory inputs (segregation), maintaining this separation of information at a representational level (representation), and using these entities to construct new inferences, predictions, and behaviors (composition). Our analysis draws inspiration from a wealth of research in neuroscience and cognitive psychology, and surveys relevant mechanisms from the machine learning literature, to help identify a combination of inductive biases that allow symbolic information processing to emerge naturally in neural networks. We believe that a compositional approach to AI, in terms of grounded symbol-like representations, is of fundamental importance for realizing human-level generalization, and we hope that this paper may contribute towards that goal as a reference and inspiration.