Analyzing Recursiveness in Multimodal Generative Artificial Intelligence: Stability or Divergence?

TL;DR

This study investigates whether recursive modality changes in multimodal AI tools like GPT-4o and DALL-E 3 lead to content divergence or stability, revealing divergence tendencies and influencing future content generation practices.

Contribution

It provides an empirical analysis of recursive modality transformations in multimodal AI, highlighting divergence behaviors and their dependence on initial content and model settings.

Findings

Recursive modality changes tend to diverge from the original content.

Divergence varies with initial image type and model configuration.

Multimodal loops do not converge to stable content, indicating potential limitations.

Abstract

One of the latest trends in generative Artificial Intelligence is tools that generate and analyze content in different modalities, such as text and images, and convert information from one to the other. From a conceptual point of view, it is interesting to study whether these modality changes incur information loss and to what extent. This is analogous to variants of the classical game telephone, where players alternate between describing images and creating drawings based on those descriptions leading to unexpected transformations of the original content. In the case of AI, modality changes can be applied recursively, starting from an image to extract a text that describes it; using the text to generate a second image, extracting a text that describes it, and so on. As this process is applied recursively, AI tools are generating content from one mode to use them to create content in another mode and so on. Ideally, the embeddings of all of them would remain close to those of the original content so that only small variations are observed in the generated content versus the original one. However, it may also be the case the distance to the original embeddings increases in each iteration leading to a divergence in the process and to content that is barely related to the original one. In this paper, we present the results of an empirical study on the impact of recursive modality changes using GPT-4o, a state-of-the-art AI multimodal tool, and DALL-E 3. The results show that the multimodality loop diverges from the initial image without converging to anything specific. We have observed differences depending on the type of initial image and the configuration of the models. These findings are particularly relevant due to the increasing use of these tools for content generation, reconstruction, and adaptation, and their potential implications for the content on the Internet of the future.