SHYI: Action Support for Contrastive Learning in High-Fidelity Text-to-Image Generation

TL;DR

This paper introduces SHYI, a novel method enhancing text-to-image generation by improving action depiction involving multiple objects through advanced contrastive learning techniques and modifications to Stable Diffusion.

Contribution

The paper proposes semantically hypergraphic contrastive adjacency learning and InteractDiffusion to significantly improve action representation in text-to-image models.

Findings

Enhanced image-text similarity scores with CLIP and TIFA

Improved depiction of multi-object actions in generated images

Promising results even with poorly understood verbs

Abstract

In this project, we address the issue of infidelity in text-to-image generation, particularly for actions involving multiple objects. For this we build on top of the CONFORM framework which uses Contrastive Learning to improve the accuracy of the generated image for multiple objects. However the depiction of actions which involves multiple different object has still large room for improvement. To improve, we employ semantically hypergraphic contrastive adjacency learning, a comprehension of enhanced contrastive structure and "contrast but link" technique. We further amend Stable Diffusion's understanding of actions by InteractDiffusion. As evaluation metrics we use image-text similarity CLIP and TIFA. In addition, we conducted a user study. Our method shows promising results even with verbs that Stable Diffusion understands mediocrely. We then provide future directions by analyzing the results. Our codebase can be found on polybox under the link: https://polybox.ethz.ch/index.php/s/dJm3SWyRohUrFxn