UMFA: A photorealistic style transfer method based on U-Net and multi-layer feature aggregation

TL;DR

This paper introduces UMFA, a photorealistic style transfer network that uses U-Net and multi-layer feature aggregation to produce natural, detailed stylized images with minimal distortion, outperforming existing methods.

Contribution

The paper presents a novel U-Net based framework with multi-layer feature aggregation and adaptive normalization for improved photorealistic style transfer without post-processing.

Findings

Achieves higher structural similarity and lower style loss.

Preserves rich content details without masks or post-processing.

Produces more photorealistic stylized images.

Abstract

In this paper, we propose a photorealistic style transfer network to emphasize the natural effect of photorealistic image stylization. In general, distortion of the image content and lacking of details are two typical issues in the style transfer field. To this end, we design a novel framework employing the U-Net structure to maintain the rich spatial clues, with a multi-layer feature aggregation (MFA) method to simultaneously provide the details obtained by the shallow layers in the stylization processing. In particular, an encoder based on the dense block and a decoder form a symmetrical structure of U-Net are jointly staked to realize an effective feature extraction and image reconstruction. Besides, a transfer module based on MFA and "adaptive instance normalization" (AdaIN) is inserted in the skip connection positions to achieve the stylization. Accordingly, the stylized image possesses the texture of a real photo and preserves rich content details without introducing any mask or post-processing steps. The experimental results on public datasets demonstrate that our method achieves a more faithful structural similarity with a lower style loss, reflecting the effectiveness and merit of our approach.