Improving Masked Style Transfer using Blended Partial Convolution

TL;DR

This paper introduces a partial-convolution-based neural network for targeted style transfer that accurately applies styles to specific image regions, improving over traditional masking methods.

Contribution

It proposes a novel partial convolution approach and internal blending techniques for precise regional style transfer, addressing limitations of existing masking methods.

Findings

Improved visual quality of region-specific style transfer

Quantitative metrics show enhanced style accuracy

Code implementation is publicly available

Abstract

Artistic style transfer has long been possible with the advancements of convolution- and transformer-based neural networks. Most algorithms apply the artistic style transfer to the whole image, but individual users may only need to apply a style transfer to a specific region in the image. The standard practice is to simply mask the image after the stylization. This work shows that this approach tends to improperly capture the style features in the region of interest. We propose a partial-convolution-based style transfer network that accurately applies the style features exclusively to the region of interest. Additionally, we present network-internal blending techniques that account for imperfections in the region selection. We show that this visually and quantitatively improves stylization using examples from the SA-1B dataset. Code is publicly available at https://github.com/davidmhart/StyleTransferMasked.