Discrete and continuous representations and processing in deep learning: Looking forward

TL;DR

This paper discusses the importance of integrating discrete symbols with continuous representations in deep learning to enhance reasoning, generalization, and intelligent behavior in machine systems.

Contribution

It advocates for combining discrete and continuous representations in neural networks and explores potential avenues for incorporating discrete elements to improve AI capabilities.

Findings

Discrete symbols facilitate abstract reasoning and knowledge composition.

Combining discrete and continuous representations can enhance generalization.

Inclusion of discrete elements may lead to more human-like intelligence in AI.

Abstract

Discrete and continuous representations of content (e.g., of language or images) have interesting properties to be explored for the understanding of or reasoning with this content by machines. This position paper puts forward our opinion on the role of discrete and continuous representations and their processing in the deep learning field. Current neural network models compute continuous-valued data. Information is compressed into dense, distributed embeddings. By stark contrast, humans use discrete symbols in their communication with language. Such symbols represent a compressed version of the world that derives its meaning from shared contextual information. Additionally, human reasoning involves symbol manipulation at a cognitive level, which facilitates abstract reasoning, the composition of knowledge and understanding, generalization and efficient learning. Motivated by these insights, in this paper we argue that combining discrete and continuous representations and their processing will be essential to build systems that exhibit a general form of intelligence. We suggest and discuss several avenues that could improve current neural networks with the inclusion of discrete elements to combine the advantages of both types of representations.