F2IDiff: Real-world Image Super-resolution using Feature to Image Diffusion Foundation Model

TL;DR

This paper introduces F2IDiff, a super-resolution method leveraging lower-level features from DINOv2 to improve image quality without hallucinations, addressing limitations of text-based conditioning in real-world smartphone images.

Contribution

The paper proposes a novel SISR approach using DINOv2 features for conditioning, enhancing detail preservation and reducing hallucinations in high-fidelity smartphone images.

Findings

Improved super-resolution quality with fewer hallucinations.

Effective use of lower-level features for image conditioning.

Better preservation of subtle textures in super-resolved images.

Abstract

With the advent of Generative AI, Single Image Super-Resolution (SISR) quality has seen substantial improvement, as the strong priors learned by Text-2-Image Diffusion (T2IDiff) Foundation Models (FM) can bridge the gap between High-Resolution (HR) and Low-Resolution (LR) images. However, flagship smartphone cameras have been slow to adopt generative models because strong generation can lead to undesirable hallucinations. For substantially degraded LR images, as seen in academia, strong generation is required and hallucinations are more tolerable because of the wide gap between LR and HR images. In contrast, in consumer photography, the LR image has substantially higher fidelity, requiring only minimal hallucination-free generation. We hypothesize that generation in SISR is controlled by the stringency and richness of the FM's conditioning feature. First, text features are high level features, which often cannot describe subtle textures in an image. Additionally, Smartphone LR images are at least $12MP$, whereas SISR networks built on T2IDiff FM are designed to perform inference on much smaller images ($<1MP$). As a result, SISR inference has to be performed on small patches, which often cannot be accurately described by text feature. To address these shortcomings, we introduce an SISR network built on a FM with lower-level feature conditioning, specifically DINOv2 features, which we call a Feature-to-Image Diffusion (F2IDiff) Foundation Model (FM). Lower level features provide stricter conditioning while being rich descriptors of even small patches.