A Scaling Law for Synthetic-to-Real Transfer: How Much Is Your Pre-training Effective?

TL;DR

This paper introduces a scaling law that predicts the effectiveness of synthetic pre-training for real vision tasks, helping determine whether to increase data or improve synthesis methods.

Contribution

It derives a simple, predictive scaling law for synthetic-to-real transfer performance and validates it across multiple experimental settings.

Findings

The scaling law accurately predicts transfer performance.

Increasing synthetic data size has diminishing returns when domain gap is large.

The generalization bound aligns with empirical results.

Abstract

Synthetic-to-real transfer learning is a framework in which a synthetically generated dataset is used to pre-train a model to improve its performance on real vision tasks. The most significant advantage of using synthetic images is that the ground-truth labels are automatically available, enabling unlimited expansion of the data size without human cost. However, synthetic data may have a huge domain gap, in which case increasing the data size does not improve the performance. How can we know that? In this study, we derive a simple scaling law that predicts the performance from the amount of pre-training data. By estimating the parameters of the law, we can judge whether we should increase the data or change the setting of image synthesis. Further, we analyze the theory of transfer learning by considering learning dynamics and confirm that the derived generalization bound is consistent with our empirical findings. We empirically validated our scaling law on various experimental settings of benchmark tasks, model sizes, and complexities of synthetic images.