Dequantization and Color Transfer with Diffusion Models

TL;DR

This paper introduces a diffusion-based method for image dequantization and color transfer that allows for controlled, plausible edits respecting user-specified palettes and textures, outperforming existing restoration techniques.

Contribution

The paper presents a novel diffusion model operating on quantized images, enabling effective palette transfer, recoloring, and texture-conditioned editing, addressing limitations of prior image restoration methods.

Findings

Model generates natural images respecting user-defined color palettes.

Proposed bipartite matching method effectively transfers color palettes.

Texture conditioning improves recoloring and maintains image realism.

Abstract

We demonstrate an image dequantizing diffusion model that enables novel edits on natural images. We propose operating on quantized images because they offer easy abstraction for patch-based edits and palette transfer. In particular, we show that color palettes can make the output of the diffusion model easier to control and interpret. We first establish that existing image restoration methods are not sufficient, such as JPEG noise reduction models. We then demonstrate that our model can generate natural images that respect the color palette the user asked for. For palette transfer, we propose a method based on weighted bipartite matching. We then show that our model generates plausible images even after extreme palette transfers, respecting user query. Our method can optionally condition on the source texture in part or all of the image. In doing so, we overcome a common problem in existing image colorization methods that are unable to produce colors with a different luminance than the input. We evaluate several possibilities for texture conditioning and their trade-offs, including luminance, image gradients, and thresholded gradients, the latter of which performed best in maintaining texture and color control simultaneously. Our method can be usefully extended to another practical edit: recoloring patches of an image while respecting the source texture. Our procedure is supported by several qualitative and quantitative evaluations.