RefSTAR: Blind Facial Image Restoration with Reference Selection, Transfer, and Reconstruction

TL;DR

RefSTAR introduces a novel method for blind facial image restoration that effectively incorporates high-quality reference images through selection, transfer, and reconstruction modules, significantly improving identity preservation and feature transfer.

Contribution

The paper proposes a new blind facial image restoration approach with a reference selection module, a feature fusion paradigm, and a reference reconstruction mechanism, addressing prior identity preservation issues.

Findings

Superior identity preservation demonstrated in experiments

Effective reference feature transfer achieved

Outperforms existing methods on various backbone models

Abstract

Blind facial image restoration is highly challenging due to unknown complex degradations and the sensitivity of humans to faces. Although existing methods introduce auxiliary information from generative priors or high-quality reference images, they still struggle with identity preservation problems, mainly due to improper feature introduction on detailed textures. In this paper, we focus on effectively incorporating appropriate features from high-quality reference images, presenting a novel blind facial image restoration method that considers reference selection, transfer, and reconstruction (RefSTAR). In terms of selection, we construct a reference selection (RefSel) module. For training the RefSel module, we construct a RefSel-HQ dataset through a mask generation pipeline, which contains annotating masks for 10,000 ground truth-reference pairs. As for the transfer, due to the trivial solution in vanilla cross-attention operations, a feature fusion paradigm is designed to force the features from the reference to be integrated. Finally, we propose a reference image reconstruction mechanism that further ensures the presence of reference image features in the output image. The cycle consistency loss is also redesigned in conjunction with the mask. Extensive experiments on various backbone models demonstrate superior performance, showing better identity preservation ability and reference feature transfer quality. Source code, dataset, and pre-trained models are available at https://github.com/yinzhicun/RefSTAR.