Golden Ratio Weighting Prevents Model Collapse

TL;DR

This paper introduces a novel weighting scheme based on the golden ratio to prevent model collapse in recursive generative model training, balancing synthetic and real data for optimal performance.

Contribution

It provides a theoretical framework and practical validation for a golden ratio-based weighting scheme to improve training stability and performance in generative models.

Findings

Optimal weighting asymptotically follows a unified expression.

In some cases, the optimal weight equals the reciprocal of the golden ratio.

Validated results on simulated and real datasets support the theory.

Abstract

Recent studies identified an intriguing phenomenon in recursive generative model training known as model collapse, where models trained on data generated by previous models exhibit severe performance degradation. Addressing this issue and developing more effective training strategies have become central challenges in generative model research. In this paper, we investigate this phenomenon within a novel framework, where generative models are iteratively trained on a combination of newly collected real data and synthetic data from the previous training step. To develop an optimal training strategy for integrating real and synthetic data, we evaluate the performance of a weighted training scheme in various scenarios, including Gaussian distribution estimation, generalized linear models, and nonparametric estimation. We theoretically characterize the impact of the mixing proportion and weighting scheme of synthetic data on the final model's performance. Our key finding is that, across different settings, the optimal weighting scheme under different proportions of synthetic data asymptotically follows a unified expression, revealing a fundamental trade-off between leveraging synthetic data and model performance. In some cases, the optimal weight assigned to real data corresponds to the reciprocal of the golden ratio. Finally, we validate our theoretical results on extensive simulated datasets and a real tabular dataset.