Oracle Guided Image Synthesis with Relative Queries

TL;DR

This paper introduces a method for user-guided image synthesis that leverages relative preference queries to control specific features in generated images, utilizing a Conditional VAE to interpret user preferences.

Contribution

It presents a novel framework combining relative queries with a Conditional VAE to isolate and control preference-relevant features in image generation.

Findings

Effective in isolating user-preferred features

Handles noisy preference data robustly

Enables intuitive image customization

Abstract

Isolating and controlling specific features in the outputs of generative models in a user-friendly way is a difficult and open-ended problem. We develop techniques that allow an oracle user to generate an image they are envisioning in their head by answering a sequence of relative queries of the form \textit{"do you prefer image $a$ or image $b$?"} Our framework consists of a Conditional VAE that uses the collected relative queries to partition the latent space into preference-relevant features and non-preference-relevant features. We then use the user's responses to relative queries to determine the preference-relevant features that correspond to their envisioned output image. Additionally, we develop techniques for modeling the uncertainty in images' predicted preference-relevant features, allowing our framework to generalize to scenarios in which the relative query training set contains noise.