Evaluating and Preserving High-level Fidelity in Super-Resolution

TL;DR

This paper emphasizes the importance of high-level fidelity in super-resolution models, introduces a new annotated dataset for evaluation, and demonstrates how foundation models and fidelity feedback can improve semantic and perceptual quality.

Contribution

It introduces the first annotated dataset for high-level fidelity in SR, analyzes metric correlations, and proposes fidelity-based fine-tuning to enhance SR performance.

Findings

High-level fidelity is crucial for reliable SR outputs.

Existing metrics poorly correlate with high-level fidelity.

Fidelity feedback improves semantic and perceptual quality.

Abstract

Recent image Super-Resolution (SR) models are achieving impressive effects in reconstructing details and delivering visually pleasant outputs. However, the overpowering generative ability can sometimes hallucinate and thus change the image content despite gaining high visual quality. This type of high-level change can be easily identified by humans yet not well-studied in existing low-level image quality metrics. In this paper, we establish the importance of measuring high-level fidelity for SR models as a complementary criterion to reveal the reliability of generative SR models. We construct the first annotated dataset with fidelity scores from different SR models, and evaluate how state-of-the-art (SOTA) SR models actually perform in preserving high-level fidelity. Based on the dataset, we then analyze how existing image quality metrics correlate with fidelity measurement, and further show that this high-level task can be better addressed by foundation models. Finally, by fine-tuning SR models based on our fidelity feedback, we show that both semantic fidelity and perceptual quality can be improved, demonstrating the potential value of our proposed criteria, both in model evaluation and optimization. We will release the dataset, code, and models upon acceptance.