Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation

TL;DR

This paper introduces Pick-a-Pic, a large open dataset of user preferences for text-to-image generation, and develops PickScore, a CLIP-based model that predicts human preferences better than existing metrics.

Contribution

The creation of Pick-a-Pic dataset and the development of PickScore, a preference predictor that outperforms existing evaluation metrics for text-to-image models.

Findings

PickScore correlates better with human preferences than other metrics.

PickScore achieves superhuman performance in preference prediction.

Using PickScore improves model evaluation and ranking in text-to-image generation.

Abstract

The ability to collect a large dataset of human preferences from text-to-image users is usually limited to companies, making such datasets inaccessible to the public. To address this issue, we create a web app that enables text-to-image users to generate images and specify their preferences. Using this web app we build Pick-a-Pic, a large, open dataset of text-to-image prompts and real users' preferences over generated images. We leverage this dataset to train a CLIP-based scoring function, PickScore, which exhibits superhuman performance on the task of predicting human preferences. Then, we test PickScore's ability to perform model evaluation and observe that it correlates better with human rankings than other automatic evaluation metrics. Therefore, we recommend using PickScore for evaluating future text-to-image generation models, and using Pick-a-Pic prompts as a more relevant dataset than MS-COCO. Finally, we demonstrate how PickScore can enhance existing text-to-image models via ranking.