Inference-Time Loss-Guided Colour Preservation in Diffusion Sampling

TL;DR

This paper introduces a training-free, inference-time method for precise color control in diffusion models, combining region-based inpainting, background re-imposition, and gradient-guided latent nudging to meet explicit color targets.

Contribution

It proposes a novel, distribution-aware, inference-time color preservation technique that does not require additional training and can be integrated into existing diffusion pipelines.

Findings

Effective color adherence in diffusion sampling

Prevents perceptual local failures in color control

Compatible with standard inpainting pipelines

Abstract

Precise color control remains a persistent failure mode in text-to-image diffusion systems, particularly in design-oriented workflows where outputs must satisfy explicit, user-specified color targets. We present an inference-time, region-constrained color preservation method that steers a pretrained diffusion model without any additional training. Our approach combines (i) ROI-based inpainting for spatial selectivity, (ii) background-latent re-imposition to prevent color drift outside the ROI, and (iii) latent nudging via gradient guidance using a composite loss defined in CIE Lab and linear RGB. The loss is constructed to control not only the mean ROI color but also the tail of the pixelwise error distribution through CVaR-style and soft-maximum penalties, with a late-start gate and a time-dependent schedule to stabilize guidance across denoising steps. We show that mean-only baselines can satisfy average color constraints while producing perceptually salient local failures, motivating our distribution-aware objective. The resulting method provides a practical, training-free mechanism for targeted color adherence that can be integrated into standard Stable Diffusion inpainting pipelines.