Inductive Biases for Deep Learning of Higher-Level Cognition

TL;DR

This paper explores the inductive biases underlying higher-level cognition in humans and animals, aiming to inform AI development by understanding principles that enable flexible and systematic generalization.

Contribution

It identifies and analyzes a broader set of inductive biases related to higher-level and sequential processing, proposing their role in advancing AI and neuroscience theories.

Findings

Deep learning exploits several key inductive biases.

Studying these biases can clarify principles of intelligence.

Insights could improve AI's out-of-distribution generalization.

Abstract

A fascinating hypothesis is that human and animal intelligence could be explained by a few principles (rather than an encyclopedic list of heuristics). If that hypothesis was correct, we could more easily both understand our own intelligence and build intelligent machines. Just like in physics, the principles themselves would not be sufficient to predict the behavior of complex systems like brains, and substantial computation might be needed to simulate human-like intelligence. This hypothesis would suggest that studying the kind of inductive biases that humans and animals exploit could help both clarify these principles and provide inspiration for AI research and neuroscience theories. Deep learning already exploits several key inductive biases, and this work considers a larger list, focusing on those which concern mostly higher-level and sequential conscious processing. The objective of clarifying these particular principles is that they could potentially help us build AI systems benefiting from humans' abilities in terms of flexible out-of-distribution and systematic generalization, which is currently an area where a large gap exists between state-of-the-art machine learning and human intelligence.