LLV-FSR: Exploiting Large Language-Vision Prior for Face Super-resolution

TL;DR

This paper introduces LLV-FSR, a face super-resolution framework that leverages large vision-language models and pluralistic priors like captions and depth maps to enhance reconstruction quality and perceptual realism.

Contribution

The novel integration of vision-language priors into face super-resolution to utilize higher-order semantic and non-visual information for improved results.

Findings

Surpasses SOTA by 0.43dB PSNR on MMCelebA-HQ dataset

Significantly improves both reconstruction and perceptual quality

Effectively incorporates pluralistic priors like captions and depth maps

Abstract

Existing face super-resolution (FSR) methods have made significant advancements, but they primarily super-resolve face with limited visual information, original pixel-wise space in particular, commonly overlooking the pluralistic clues, like the higher-order depth and semantics, as well as non-visual inputs (text caption and description). Consequently, these methods struggle to produce a unified and meaningful representation from the input face. We suppose that introducing the language-vision pluralistic representation into unexplored potential embedding space could enhance FSR by encoding and exploiting the complementarity across language-vision prior. This motivates us to propose a new framework called LLV-FSR, which marries the power of large vision-language model and higher-order visual prior with the challenging task of FSR. Specifically, besides directly absorbing knowledge from original input, we introduce the pre-trained vision-language model to generate pluralistic priors, involving the image caption, descriptions, face semantic mask and depths. These priors are then employed to guide the more critical feature representation, facilitating realistic and high-quality face super-resolution. Experimental results demonstrate that our proposed framework significantly improves both the reconstruction quality and perceptual quality, surpassing the SOTA by 0.43dB in terms of PSNR on the MMCelebA-HQ dataset.