Class-Conditional self-reward mechanism for improved Text-to-Image models

TL;DR

This paper introduces a class-conditional self-reward mechanism for Text-to-Image models, enhancing image quality, automation, and prompt adherence by fine-tuning diffusion models with self-generated data and auxiliary pre-trained models.

Contribution

It presents a novel self-rewarding approach for Text-to-Image models, improving quality and automation through fine-tuning with self-judged data conditioned on object sets.

Findings

Performance improved by at least 60% over existing models.

Automated image generation with higher visual quality.

Enhanced adherence to prompt instructions.

Abstract

Self-rewarding have emerged recently as a powerful tool in the field of Natural Language Processing (NLP), allowing language models to generate high-quality relevant responses by providing their own rewards during training. This innovative technique addresses the limitations of other methods that rely on human preferences. In this paper, we build upon the concept of self-rewarding models and introduce its vision equivalent for Text-to-Image generative AI models. This approach works by fine-tuning diffusion model on a self-generated self-judged dataset, making the fine-tuning more automated and with better data quality. The proposed mechanism makes use of other pre-trained models such as vocabulary based-object detection, image captioning and is conditioned by the a set of object for which the user might need to improve generated data quality. The approach has been implemented, fine-tuned and evaluated on stable diffusion and has led to a performance that has been evaluated to be at least 60\% better than existing commercial and research Text-to-image models. Additionally, the built self-rewarding mechanism allowed a fully automated generation of images, while increasing the visual quality of the generated images and also more efficient following of prompt instructions. The code used in this work is freely available on https://github.com/safouaneelg/SRT2I.