Test-Time Adaptation for Super-Resolution: You Only Need to Overfit on a Few More Images

TL;DR

This paper introduces a simple test-time adaptation method for super-resolution that fine-tunes a pre-trained network on a few similar images, improving perceptual quality while maintaining fidelity metrics.

Contribution

It proposes a universal fine-tuning approach based on activation similarity, enhancing perceptual quality without sacrificing PSNR/SSIM, and analyzes filter correlations for better understanding.

Findings

Fine-tuning improves perceptual quality with minimal PSNR/SSIM loss.

The adapted network's filters are closer to 'ideal' filters than baseline.

Method is effective without requiring high-resolution reference images.

Abstract

Existing reference (RF)-based super-resolution (SR) models try to improve perceptual quality in SR under the assumption of the availability of high-resolution RF images paired with low-resolution (LR) inputs at testing. As the RF images should be similar in terms of content, colors, contrast, etc. to the test image, this hinders the applicability in a real scenario. Other approaches to increase the perceptual quality of images, including perceptual loss and adversarial losses, tend to dramatically decrease fidelity to the ground-truth through significant decreases in PSNR/SSIM. Addressing both issues, we propose a simple yet universal approach to improve the perceptual quality of the HR prediction from a pre-trained SR network on a given LR input by further fine-tuning the SR network on a subset of images from the training dataset with similar patterns of activation as the initial HR prediction, with respect to the filters of a feature extractor. In particular, we show the effects of fine-tuning on these images in terms of the perceptual quality and PSNR/SSIM values. Contrary to perceptually driven approaches, we demonstrate that the fine-tuned network produces a HR prediction with both greater perceptual quality and minimal changes to the PSNR/SSIM with respect to the initial HR prediction. Further, we present novel numerical experiments concerning the filters of SR networks, where we show through filter correlation, that the filters of the fine-tuned network from our method are closer to "ideal" filters, than those of the baseline network or a network fine-tuned on random images.