The Foundations of Deep Learning with a Path Towards General Intelligence

TL;DR

This paper discusses the foundational principles of deep learning, reviews its successes and limitations, and proposes directions for developing general-purpose AI with insights from neuroscience.

Contribution

It formulates axiomatic requirements for human-level AI, analyzes deep learning assumptions, and suggests extensions to achieve general intelligence.

Findings

Deep learning combines model richness, generality, and practicality.

It has achieved outstanding results through function approximation and back-propagation.

Limitations and failure modes of deep learning are identified.

Abstract

Like any field of empirical science, AI may be approached axiomatically. We formulate requirements for a general-purpose, human-level AI system in terms of postulates. We review the methodology of deep learning, examining the explicit and tacit assumptions in deep learning research. Deep Learning methodology seeks to overcome limitations in traditional machine learning research as it combines facets of model richness, generality, and practical applicability. The methodology so far has produced outstanding results due to a productive synergy of function approximation, under plausible assumptions of irreducibility and the efficiency of back-propagation family of algorithms. We examine these winning traits of deep learning, and also observe the various known failure modes of deep learning. We conclude by giving recommendations on how to extend deep learning methodology to cover the postulates of general-purpose AI including modularity, and cognitive architecture. We also relate deep learning to advances in theoretical neuroscience research.