MAGIC: Mask-Guided Image Synthesis by Inverting a Quasi-Robust Classifier

TL;DR

MAGIC introduces a mask-guided image synthesis method that inverts a quasi-robust classifier, enabling precise control over shape, location, and deformations in generated images with high fidelity and user control.

Contribution

The paper presents MAGIC, a novel one-shot mask-guided image synthesis technique that leverages structured gradients from a quasi-robust classifier for improved semantic preservation and control.

Findings

Achieves shape and location control in image synthesis.

Supports intense non-rigid shape deformations.

Provides firm user control via binary guide masks.

Abstract

We offer a method for one-shot mask-guided image synthesis that allows controlling manipulations of a single image by inverting a quasi-robust classifier equipped with strong regularizers. Our proposed method, entitled MAGIC, leverages structured gradients from a pre-trained quasi-robust classifier to better preserve the input semantics while preserving its classification accuracy, thereby guaranteeing credibility in the synthesis. Unlike current methods that use complex primitives to supervise the process or use attention maps as a weak supervisory signal, MAGIC aggregates gradients over the input, driven by a guide binary mask that enforces a strong, spatial prior. MAGIC implements a series of manipulations with a single framework achieving shape and location control, intense non-rigid shape deformations, and copy/move operations in the presence of repeating objects and gives users firm control over the synthesis by requiring to simply specify binary guide masks. Our study and findings are supported by various qualitative comparisons with the state-of-the-art on the same images sampled from ImageNet and quantitative analysis using machine perception along with a user survey of 100+ participants that endorse our synthesis quality. Project page at https://mozhdehrouhsedaghat.github.io/magic.html. Code is available at https://github.com/mozhdehrouhsedaghat/magic