The Relativity of Induction

TL;DR

This paper introduces relativistic principles of learning, emphasizing the contingent and relative nature of simplicity and generalization, offering new insights into why deep learning often outperforms traditional theoretical expectations.

Contribution

It proposes a set of relativistic principles that challenge classical views on simplicity and generalization, providing a new framework for understanding learning processes including deep learning.

Findings

Concepts of simplicity are fundamentally contingent.

All learning operates relative to an initial guess.

Generalization can be expected with sufficient observation.

Abstract

Lately there has been a lot of discussion about why deep learning algorithms perform better than we would theoretically suspect. To get insight into this question, it helps to improve our understanding of how learning works. We explore the core problem of generalization and show that long-accepted Occam's razor and parsimony principles are insufficient to ground learning. Instead, we derive and demonstrate a set of relativistic principles that yield clearer insight into the nature and dynamics of learning. We show that concepts of simplicity are fundamentally contingent, that all learning operates relative to an initial guess, and that generalization cannot be measured or strongly inferred, but that it can be expected given enough observation. Using these principles, we reconstruct our understanding in terms of distributed learning systems whose components inherit beliefs and update them. We then apply this perspective to elucidate the nature of some real world inductive processes including deep learning.