On the Origin of Species of Self-Supervised Learning

TL;DR

This paper explores the evolutionary origins and diversification principles of self-supervised learning systems, proposing a unifying theory and emphasizing the importance of digital biodiversity.

Contribution

It introduces a framework for understanding self-supervised learning as an evolving species, including a catalog of models, mutation mechanisms, and a new unifying theory.

Findings

Established a new state-of-the-art unifying theory of self-supervised learning

Cataloged diverse self-supervised models with heritable features

Compared the theory against existing benchmarks

Abstract

In the quiet backwaters of cs.CV, cs.LG and stat.ML, a cornucopia of new learning systems is emerging from a primordial soup of mathematics-learning systems with no need for external supervision. To date, little thought has been given to how these self-supervised learners have sprung into being or the principles that govern their continuing diversification. After a period of deliberate study and dispassionate judgement during which each author set their Zoom virtual background to a separate Galapagos island, we now entertain no doubt that each of these learning machines are lineal descendants of some older and generally extinct species. We make five contributions: (1) We gather and catalogue row-major arrays of machine learning specimens, each exhibiting heritable discriminative features; (2) We document a mutation mechanism by which almost imperceptible changes are introduced to the genotype of new systems, but their phenotype (birdsong in the form of tweets and vestigial plumage such as press releases) communicates dramatic changes; (3) We propose a unifying theory of self-supervised machine evolution and compare to other unifying theories on standard unifying theory benchmarks, where we establish a new (and unifying) state of the art; (4) We discuss the importance of digital biodiversity, in light of the endearingly optimistic Paris Agreement.