Contrastive Learning and Abstract Concepts: The Case of Natural Numbers

TL;DR

This paper explores applying contrastive learning to the abstract concept of natural numbers, demonstrating its effectiveness in counting tasks and superior generalization compared to supervised learning in distribution-shift scenarios.

Contribution

It introduces a novel application of contrastive learning to an abstract concept, natural numbers, and shows its advantages over supervised learning in generalization.

Findings

Contrastive learning achieves high accuracy in counting tasks.

CL outperforms supervised learning in distribution-shift scenarios.

Both methods perform similarly on baseline experiments.

Abstract

Contrastive Learning (CL) has been successfully applied to classification and other downstream tasks related to concrete concepts, such as objects contained in the ImageNet dataset. No attempts seem to have been made so far in applying this promising scheme to more abstract entities. A prominent example of these could be the concept of (discrete) Quantity. CL can be frequently interpreted as a self-supervised scheme guided by some profound and ubiquitous conservation principle (e.g. conservation of identity in object classification tasks). In this introductory work we apply a suitable conservation principle to the semi-abstract concept of natural numbers by which discrete quantities can be estimated or predicted. We experimentally show, by means of a toy problem, that contrastive learning can be trained to count at a glance with high accuracy both at human as well as at super-human ranges.. We compare this with the results of a trained-to-count at a glance supervised learning (SL) neural network scheme of similar architecture. We show that both schemes exhibit similar good performance on baseline experiments, where the distributions of the training and testing stages are equal. Importantly, we demonstrate that in some generalization scenarios, where training and testing distributions differ, CL boasts more robust and much better error performance.