Color and Texture Dual Pipeline Lightweight Style Transfer

TL;DR

This paper introduces a lightweight dual pipeline style transfer method that separately handles color and texture, reducing artifacts and enabling controllable texture addition, with state-of-the-art results and minimal model size.

Contribution

The proposed CTDP method employs a dual pipeline for simultaneous color and texture transfer, introducing a masked total variation loss and controllable texture addition, with significantly smaller model size.

Findings

Achieves state-of-the-art style transfer quality.

Model size is as low as 20k, much smaller than existing models.

Effectively suppresses artifacts and enables controllable texture enhancement.

Abstract

Style transfer methods typically generate a single stylized output of color and texture coupling for reference styles, and color transfer schemes may introduce distortion or artifacts when processing reference images with duplicate textures. To solve the problem, we propose a Color and Texture Dual Pipeline Lightweight Style Transfer CTDP method, which employs a dual pipeline method to simultaneously output the results of color and texture transfer. Furthermore, we designed a masked total variation loss to suppress artifacts and small texture representations in color transfer results without affecting the semantic part of the content. More importantly, we are able to add texture structures with controllable intensity to color transfer results for the first time. Finally, we conducted feature visualization analysis on the texture generation mechanism of the framework and found that smoothing the input image can almost completely eliminate this texture structure. In comparative experiments, the color and texture transfer results generated by CTDP both achieve state-of-the-art performance. Additionally, the weight of the color transfer branch model size is as low as 20k, which is 100-1500 times smaller than that of other state-of-the-art models.