Test-time Cost-and-Quality Controllable Arbitrary-Scale Super-Resolution with Variable Fourier Components

TL;DR

This paper introduces a test-time controllable super-resolution method using an RNN with Fourier components, allowing dynamic adjustment of quality and computational cost without retraining.

Contribution

The proposed method enables test-time control of SR quality and cost via Fourier components estimated by an RNN, eliminating the need for model retraining.

Findings

More Fourier components improve PSNR.

Fewer Fourier components still outperform other methods in PSNR drop.

The method allows dynamic quality-cost trade-off at test time.

Abstract

Super-resolution (SR) with arbitrary scale factor and cost-and-quality controllability at test time is essential for various applications. While several arbitrary-scale SR methods have been proposed, these methods require us to modify the model structure and retrain it to control the computational cost and SR quality. To address this limitation, we propose a novel SR method using a Recurrent Neural Network (RNN) with the Fourier representation. In our method, the RNN sequentially estimates Fourier components, each consisting of frequency and amplitude, and aggregates these components to reconstruct an SR image. Since the RNN can adjust the number of recurrences at test time, we can control the computational cost and SR quality in a single model: fewer recurrences (i.e., fewer Fourier components) lead to lower cost but lower quality, while more recurrences (i.e., more Fourier components) lead to better quality but more cost. Experimental results prove that more Fourier components improve the PSNR score. Furthermore, even with fewer Fourier components, our method achieves a lower PSNR drop than other state-of-the-art arbitrary-scale SR methods.