Universal Photorealistic Style Transfer: A Lightweight and Adaptive Approach

TL;DR

This paper introduces UPST, a lightweight, pretraining-free framework for high-resolution photorealistic style transfer on images and videos, emphasizing efficiency, accuracy, and preservation of realism.

Contribution

The proposed UPST framework offers a novel, adaptive, and efficient approach to photorealistic style transfer without pretraining, suitable for high-resolution images and videos.

Findings

Produces photorealistic outputs with high fidelity.

Reduces GPU memory usage significantly.

Supports high-resolution and video style transfer without additional constraints.

Abstract

Photorealistic style transfer aims to apply stylization while preserving the realism and structure of input content. However, existing methods often encounter challenges such as color tone distortions, dependency on pair-wise pre-training, inefficiency with high-resolution inputs, and the need for additional constraints in video style transfer tasks. To address these issues, we propose a Universal Photorealistic Style Transfer (UPST) framework that delivers accurate photorealistic style transfer on high-resolution images and videos without relying on pre-training. Our approach incorporates a lightweight StyleNet for per-instance transfer, ensuring color tone accuracy while supporting high-resolution inputs, maintaining rapid processing speeds, and eliminating the need for pretraining. To further enhance photorealism and efficiency, we introduce instance-adaptive optimization, which features an adaptive coefficient to prioritize content image realism and employs early stopping to accelerate network convergence. Additionally, UPST enables seamless video style transfer without additional constraints due to its strong non-color information preservation ability. Experimental results show that UPST consistently produces photorealistic outputs and significantly reduces GPU memory usage, making it an effective and universal solution for various photorealistic style transfer tasks.