MADI: Masking-Augmented Diffusion with Inference-Time Scaling for Visual Editing

TL;DR

MADI enhances diffusion models for visual editing by introducing a novel training strategy and inference-time scaling, significantly improving their controllability, compositionality, and editability in image generation tasks.

Contribution

The paper proposes Masking-Augmented Diffusion with Inference-Time Scaling (MADI), combining a new training method and inference mechanism to improve structured visual editing capabilities.

Findings

Enhanced editability and controllability of diffusion models.

Improved performance with expressive prompts during training.

Significant gains in localized and structure-aware image editing.

Abstract

Despite the remarkable success of diffusion models in text-to-image generation, their effectiveness in grounded visual editing and compositional control remains challenging. Motivated by advances in self-supervised learning and in-context generative modeling, we propose a series of simple yet powerful design choices that significantly enhance diffusion model capacity for structured, controllable generation and editing. We introduce Masking-Augmented Diffusion with Inference-Time Scaling (MADI), a framework that improves the editability, compositionality and controllability of diffusion models through two core innovations. First, we introduce Masking-Augmented gaussian Diffusion (MAgD), a novel training strategy with dual corruption process which combines standard denoising score matching and masked reconstruction by masking noisy input from forward process. MAgD encourages the model to learn discriminative and compositional visual representations, thus enabling localized and structure-aware editing. Second, we introduce an inference-time capacity scaling mechanism based on Pause Tokens, which act as special placeholders inserted into the prompt for increasing computational capacity at inference time. Our findings show that adopting expressive and dense prompts during training further enhances performance, particularly for MAgD. Together, these contributions in MADI substantially enhance the editability of diffusion models, paving the way toward their integration into more general-purpose, in-context generative diffusion architectures.