AnyRefill: A Unified, Data-Efficient Framework for Left-Prompt-Guided Vision Tasks

TL;DR

AnyRefill introduces a unified, data-efficient framework leveraging left-prompt-guided reformulation and inpainting priors to effectively address diverse reference-based vision tasks without extra visual encoders.

Contribution

It proposes AnyRefill, a novel extension of LeftRefill, that adapts Text-to-Image models for multiple vision tasks using a left-right stitching approach and minimal fine-tuning.

Findings

Outperforms other image condition injection methods.

Achieves competitive results with state-of-the-art open-source tools.

Maintains high performance with minimal task-specific fine-tuning.

Abstract

In this paper, we present a novel Left-Prompt-Guided (LPG) paradigm to address a diverse range of reference-based vision tasks. Inspired by the human creative process, we reformulate these tasks using a left-right stitching formulation to construct contextual input. Building upon this foundation, we propose AnyRefill, an extension of LeftRefill, that effectively adapts Text-to-Image (T2I) models to various vision tasks. AnyRefill leverages the inpainting priors of advanced T2I model based on the Diffusion Transformer (DiT) architecture, and incorporates flexible components to enhance its capabilities. By combining task-specific LoRAs with the stitching input, AnyRefill unlocks its potential across diverse tasks, including conditional generation, visual perception, and image editing, without requiring additional visual encoders. Meanwhile, AnyRefill exhibits remarkable data efficiency, requiring minimal task-specific fine-tuning while maintaining high generative performance. Through extensive ablation studies, we demonstrate that AnyRefill outperforms other image condition injection methods and achieves competitive results compared to state-of-the-art open-source methods. Notably, AnyRefill delivers results comparable to advanced commercial tools, such as IC-Light and SeedEdit, even in challenging scenarios. Comprehensive experiments and ablation studies across versatile tasks validate the strong generation of the proposed simple yet effective LPG formulation, establishing AnyRefill as a unified, highly data-efficient solution for reference-based vision tasks.