ASteISR: Adapting Single Image Super-resolution Pre-trained Model for Efficient Stereo Image Super-resolution

TL;DR

This paper introduces a parameter-efficient fine-tuning approach for adapting pre-trained single-image super-resolution models to stereo image super-resolution, significantly reducing training resources while achieving state-of-the-art results.

Contribution

The paper proposes stereo and spatial adapters for efficient transfer learning from SISR to SteISR, enabling high performance with minimal parameter updates.

Findings

Improves stereo image inference accuracy by 0.79dB on Flickr1024

Trains only 4.8% of the original model parameters

Reduces training time by 57% and memory usage by 15%

Abstract

Despite advances in the paradigm of pre-training then fine-tuning in low-level vision tasks, significant challenges persist particularly regarding the increased size of pre-trained models such as memory usage and training time. Another concern often encountered is the unsatisfying results yielded when directly applying pre-trained single-image models to multi-image domain. In this paper, we propose a efficient method for transferring a pre-trained single-image super-resolution (SISR) transformer network to the domain of stereo image super-resolution (SteISR) through a parameter-efficient fine-tuning (PEFT) method. Specifically, we introduce the concept of stereo adapters and spatial adapters which are incorporated into the pre-trained SISR transformer network. Subsequently, the pre-trained SISR model is frozen, enabling us to fine-tune the adapters using stereo datasets along. By adopting this training method, we enhance the ability of the SISR model to accurately infer stereo images by 0.79dB on the Flickr1024 dataset. This method allows us to train only 4.8% of the original model parameters, achieving state-of-the-art performance on four commonly used SteISR benchmarks. Compared to the more complicated full fine-tuning approach, our method reduces training time and memory consumption by 57% and 15%, respectively.