Image Content Generation with Causal Reasoning

TL;DR

This paper introduces a new visual question answering with image (VQAI) task, creates a dataset based on Tom and Jerry, and proposes a novel image generation paradigm to incorporate causal reasoning in visual content creation.

Contribution

It pioneers the integration of causal reasoning into visual content generation and establishes a new dataset and paradigm for VQAI tasks.

Findings

Developed a new VQAI dataset based on Tom and Jerry

Proposed a novel image generation paradigm for causal reasoning

Conducted extensive experiments demonstrating the approach's effectiveness

Abstract

The emergence of ChatGPT has once again sparked research in generative artificial intelligence (GAI). While people have been amazed by the generated results, they have also noticed the reasoning potential reflected in the generated textual content. However, this current ability for causal reasoning is primarily limited to the domain of language generation, such as in models like GPT-3. In visual modality, there is currently no equivalent research. Considering causal reasoning in visual content generation is significant. This is because visual information contains infinite granularity. Particularly, images can provide more intuitive and specific demonstrations for certain reasoning tasks, especially when compared to coarse-grained text. Hence, we propose a new image generation task called visual question answering with image (VQAI) and establish a dataset of the same name based on the classic \textit{Tom and Jerry} animated series. Additionally, we develop a new paradigm for image generation to tackle the challenges of this task. Finally, we perform extensive experiments and analyses, including visualizations of the generated content and discussions on the potentials and limitations. The code and data are publicly available under the license of CC BY-NC-SA 4.0 for academic and non-commercial usage. The code and dataset are publicly available at: https://github.com/IEIT-AGI/MIX-Shannon/blob/main/projects/VQAI/lgd_vqai.md.