Unleashing In-context Learning of Autoregressive Models for Few-shot Image Manipulation

TL;DR

This paper introduces InstaManip, a multi-modal autoregressive model that enables rapid, few-shot image manipulation guided by text and visual examples, overcoming diffusion models' reasoning limitations.

Contribution

The paper proposes a novel group self-attention mechanism and relation regularization to improve few-shot image manipulation with in-context learning.

Findings

Outperforms previous models by at least 19% in human evaluation

Model's performance improves with more diverse exemplars

Effective disentanglement of image features achieved

Abstract

Text-guided image manipulation has experienced notable advancement in recent years. In order to mitigate linguistic ambiguity, few-shot learning with visual examples has been applied for instructions that are underrepresented in the training set, or difficult to describe purely in language. However, learning from visual prompts requires strong reasoning capability, which diffusion models are struggling with. To address this issue, we introduce a novel multi-modal autoregressive model, dubbed $\textbf{InstaManip}$, that can $\textbf{insta}$ntly learn a new image $\textbf{manip}$ulation operation from textual and visual guidance via in-context learning, and apply it to new query images. Specifically, we propose an innovative group self-attention mechanism to break down the in-context learning process into two separate stages -- learning and applying, which simplifies the complex problem into two easier tasks. We also introduce a relation regularization method to further disentangle image transformation features from irrelevant contents in exemplar images. Extensive experiments suggest that our method surpasses previous few-shot image manipulation models by a notable margin ($\geq$19% in human evaluation). We also find our model can be further boosted by increasing the number or diversity of exemplar images.