Sharing the Learned Knowledge-base to Estimate Convolutional Filter Parameters for Continual Image Restoration

TL;DR

This paper introduces a simple convolution layer modification that enables continual learning in image restoration tasks, allowing models to adapt to new challenges without retraining from scratch or significant architectural changes.

Contribution

It proposes a knowledge-sharing convolutional layer modification that facilitates continual learning in image restoration, avoiding heavy architectural changes and reducing computational overhead.

Findings

Effective in adding new restoration tasks without performance loss

Improves performance on new tasks by leveraging previous knowledge

Maintains computational efficiency with increased trainable parameters

Abstract

Continual learning is an emerging topic in the field of deep learning, where a model is expected to learn continuously for new upcoming tasks without forgetting previous experiences. This field has witnessed numerous advancements, but few works have been attempted in the direction of image restoration. Handling large image sizes and the divergent nature of various degradation poses a unique challenge in the restoration domain. However, existing works require heavily engineered architectural modifications for new task adaptation, resulting in significant computational overhead. Regularization-based methods are unsuitable for restoration, as different restoration challenges require different kinds of feature processing. In this direction, we propose a simple modification of the convolution layer to adapt the knowledge from previous restoration tasks without touching the main backbone architecture. Therefore, it can be seamlessly applied to any deep architecture without any structural modifications. Unlike other approaches, we demonstrate that our model can increase the number of trainable parameters without significantly increasing computational overhead or inference time. Experimental validation demonstrates that new restoration tasks can be introduced without compromising the performance of existing tasks. We also show that performance on new restoration tasks improves by adapting the knowledge from the knowledge base created by previous restoration tasks. The code is available at https://github.com/aupendu/continual-restore.