Generating Synthetic Datasets by Interpolating along Generalized Geodesics

TL;DR

This paper introduces a novel method for synthesizing new datasets by interpolating existing datasets along generalized geodesics in optimal transport space, enabling targeted pretraining for transfer learning.

Contribution

It extends dataset synthesis to all possible combinations of datasets using generalized geodesics, including those with different label sets, via scalable optimal transport methods.

Findings

Effective dataset interpolation between unrelated label sets.

Improved transfer learning performance in computer vision tasks.

Scalable and efficient synthesis of targeted datasets.

Abstract

Data for pretraining machine learning models often consists of collections of heterogeneous datasets. Although training on their union is reasonable in agnostic settings, it might be suboptimal when the target domain -- where the model will ultimately be used -- is known in advance. In that case, one would ideally pretrain only on the dataset(s) most similar to the target one. Instead of limiting this choice to those datasets already present in the pretraining collection, here we explore extending this search to all datasets that can be synthesized as `combinations' of them. We define such combinations as multi-dataset interpolations, formalized through the notion of generalized geodesics from optimal transport (OT) theory. We compute these geodesics using a recent notion of distance between labeled datasets, and derive alternative interpolation schemes based on it: using either barycentric projections or optimal transport maps, the latter computed using recent neural OT methods. These methods are scalable, efficient, and -- notably -- can be used to interpolate even between datasets with distinct and unrelated label sets. Through various experiments in transfer learning in computer vision, we demonstrate this is a promising new approach for targeted on-demand dataset synthesis.