G2L: A Geometric Approach for Generating Pseudo-labels that Improve Transfer Learning

TL;DR

G2L introduces a geometric method using high-dimensional determinants to automatically generate pseudo-labels, enhancing transfer learning by improving model transferability without extensive human annotation.

Contribution

The paper presents a novel geometric algorithm based on the Cayley-Menger determinant for creating pseudo-labels, improving transfer learning performance across diverse datasets.

Findings

Achieves a 0.43% overall error reduction compared to human-annotated models.

Generates models with similar or better transferability in 4 out of 5 tested datasets.

Demonstrates tunability of pseudo-label quality via dataset divergence measures.

Abstract

Transfer learning is a deep-learning technique that ameliorates the problem of learning when human-annotated labels are expensive and limited. In place of such labels, it uses instead the previously trained weights from a well-chosen source model as the initial weights for the training of a base model for a new target dataset. We demonstrate a novel but general technique for automatically creating such source models. We generate pseudo-labels according to an efficient and extensible algorithm that is based on a classical result from the geometry of high dimensions, the Cayley-Menger determinant. This G2L (``geometry to label'') method incrementally builds up pseudo-labels using a greedy computation of hypervolume content. We demonstrate that the method is tunable with respect to expected accuracy, which can be forecast by an information-theoretic measure of dataset similarity (divergence) between source and target. The results of 280 experiments show that this mechanical technique generates base models that have similar or better transferability compared to a baseline of models trained on extensively human-annotated ImageNet1K labels, yielding an overall error decrease of 0.43\%, and an error decrease in 4 out of 5 divergent datasets tested.