DeFLOCNet: Deep Image Editing via Flexible Low-level Controls

TL;DR

DeFLOCNet is a deep learning model that improves image editing by directly integrating low-level user controls into the CNN's feature space, enabling more faithful and detailed content creation.

Contribution

It introduces a novel method of injecting low-level controls into each structure generation block within a deep encoder-decoder CNN for enhanced image editing fidelity.

Findings

Effective transformation of user controls into visual content

Improved fidelity in structure and texture generation

Outperforms existing methods on benchmark datasets

Abstract

User-intended visual content fills the hole regions of an input image in the image editing scenario. The coarse low-level inputs, which typically consist of sparse sketch lines and color dots, convey user intentions for content creation (\ie, free-form editing). While existing methods combine an input image and these low-level controls for CNN inputs, the corresponding feature representations are not sufficient to convey user intentions, leading to unfaithfully generated content. In this paper, we propose DeFLOCNet which relies on a deep encoder-decoder CNN to retain the guidance of these controls in the deep feature representations. In each skip-connection layer, we design a structure generation block. Instead of attaching low-level controls to an input image, we inject these controls directly into each structure generation block for sketch line refinement and color propagation in the CNN feature space. We then concatenate the modulated features with the original decoder features for structure generation. Meanwhile, DeFLOCNet involves another decoder branch for texture generation and detail enhancement. Both structures and textures are rendered in the decoder, leading to user-intended editing results. Experiments on benchmarks demonstrate that DeFLOCNet effectively transforms different user intentions to create visually pleasing content.