MAR-MAER: Metric-Aware and Ambiguity-Adaptive Autoregressive Image Generation

TL;DR

MAR-MAER is a hierarchical autoregressive framework for text-to-image generation that improves image quality and handles ambiguous prompts by aligning representations with human metrics and incorporating controlled randomness.

Contribution

It introduces a metric-aware embedding regularization and a probabilistic latent model for ambiguity, surpassing previous models in quality and diversity.

Findings

Achieves +1.6 CLIPScore over baseline

Improves HPSv2 by +5.3 points

Produces more diverse images for ambiguous prompts

Abstract

Autoregressive (AR) models have demonstrated significant success in the realm of text-to-image generation. However, they usually face two major challenges. Firstly, the generated images may not always meet the quality standards expected by humans. Furthermore, these models face difficulty when dealing with ambiguous prompts that could be interpreted in several valid ways. To address these issues, we introduce MAR-MAER, an innovative hierarchical autoregressive framework. It combines two main components. It is a metric-aware embedding regularization method. The other one is a probabilistic latent model used for handling ambiguous semantics. Our method utilizes a lightweight projection head, which is trained with an adaptive kernel regression loss function. This aligns the model's internal representations with human-preferred quality metrics, such as CLIPScore and HPSv2. As a result, the embedding space that is learned more accurately reflects human judgment. We are also introducing a conditional variational module. This approach incorporates an aspect of controlled randomness within the hierarchical token generation process. This capability allows the model to produce a diverse array of coherent images based on ambiguous or open-ended prompts. We conducted extensive experiments using COCO and a newly developed Ambiguous-Prompt Benchmark. The results show that MAR-MAER achieves excellent performance in both metric consistency and semantic flexibility. It exceeds the baseline Hi-MAR model's performance, showing an improvement of +1.6 in CLIPScore and +5.3 in HPSv2. For unclear inputs, it produces a notably wider range of outputs. These findings have been confirmed through both human evaluation and automated metrics.