Experimental evidence of progressive ChatGPT models self-convergence

TL;DR

This study provides empirical evidence that successive ChatGPT models tend to produce increasingly similar outputs over time, indicating a self-convergence phenomenon likely caused by synthetic data influence in training datasets.

Contribution

It is the first longitudinal investigation demonstrating how ChatGPT models' output diversity diminishes over successive versions due to self-convergence.

Findings

Recent ChatGPT models show reduced output diversity over time.

Synthetic data infiltration in training datasets influences model convergence.

Output similarity increases among different ChatGPT versions.

Abstract

Large Language Models (LLMs) that undergo recursive training on synthetically generated data are susceptible to model collapse, a phenomenon marked by the generation of meaningless output. Existing research has examined this issue from either theoretical or empirical perspectives, often focusing on a single model trained recursively on its own outputs. While prior studies have cautioned against the potential degradation of LLM output quality under such conditions, no longitudinal investigation has yet been conducted to assess this effect over time. In this study, we employ a text similarity metric to evaluate different ChatGPT models' capacity to generate diverse textual outputs. Our findings indicate a measurable decline of recent ChatGPT releases' ability to produce varied text, even when explicitly prompted to do so, by setting the temperature parameter to one. The observed reduction in output diversity may be attributed to the influence of the amounts of synthetic data incorporated within their training datasets as the result of internet infiltration by LLM generated data. The phenomenon is defined as model self-convergence because of the gradual increase of similarities of produced texts among different ChatGPT versions.