General Intelligence Requires Rethinking Exploration

TL;DR

This paper emphasizes the importance of exploration in AI, proposing a unified framework for exploration across supervised and reinforcement learning to advance towards more general intelligence.

Contribution

It introduces the concept of generalized exploration, unifying exploration strategies across learning paradigms to foster open-ended, continual learning for AI.

Findings

Exploration is crucial beyond reinforcement learning, including supervised learning.

Unified exploration framework highlights common challenges across learning types.

Open-ended exploration is key to achieving more general AI.

Abstract

We are at the cusp of a transition from "learning from data" to "learning what data to learn from" as a central focus of artificial intelligence (AI) research. While the first-order learning problem is not completely solved, large models under unified architectures, such as transformers, have shifted the learning bottleneck from how to effectively train our models to how to effectively acquire and use task-relevant data. This problem, which we frame as exploration, is a universal aspect of learning in open-ended domains, such as the real world. Although the study of exploration in AI is largely limited to the field of reinforcement learning, we argue that exploration is essential to all learning systems, including supervised learning. We propose the problem of generalized exploration to conceptually unify exploration-driven learning between supervised learning and reinforcement learning, allowing us to highlight key similarities across learning settings and open research challenges. Importantly, generalized exploration serves as a necessary objective for maintaining open-ended learning processes, which in continually learning to discover and solve new problems, provides a promising path to more general intelligence.