Puff-Net: Efficient Style Transfer with Pure Content and Style Feature Fusion Network

TL;DR

Puff-Net is a lightweight transformer-based style transfer network that efficiently fuses pure content and style features, achieving superior stylization quality with reduced computational costs.

Contribution

The paper introduces Puff-Net, a novel encoder-only transformer model that effectively combines content and style features for improved style transfer performance.

Findings

Outperforms state-of-the-art methods in qualitative and quantitative evaluations.

Reduces computational cost by using only an encoder transformer.

Effectively prevents under-stylization and content loss in stylized images.

Abstract

Style transfer aims to render an image with the artistic features of a style image, while maintaining the original structure. Various methods have been put forward for this task, but some challenges still exist. For instance, it is difficult for CNN-based methods to handle global information and long-range dependencies between input images, for which transformer-based methods have been proposed. Although transformers can better model the relationship between content and style images, they require high-cost hardware and time-consuming inference. To address these issues, we design a novel transformer model that includes only the encoder, thus significantly reducing the computational cost. In addition, we also find that existing style transfer methods may lead to images under-stylied or missing content. In order to achieve better stylization, we design a content feature extractor and a style feature extractor, based on which pure content and style images can be fed to the transformer. Finally, we propose a novel network termed Puff-Net, i.e., pure content and style feature fusion network. Through qualitative and quantitative experiments, we demonstrate the advantages of our model compared to state-of-the-art ones in the literature.