Cross-Stitched Multi-task Dual Recursive Networks for Unified Single Image Deraining and Desnowing

TL;DR

This paper introduces CMUDRN, a multi-task neural network that simultaneously performs single image deraining and desnowing, sharing features across tasks and enabling blind image restoration with improved or comparable performance.

Contribution

The paper proposes a novel multi-task learning framework with cross-stitch units for joint deraining and desnowing, enhancing feature sharing and enabling blind image restoration.

Findings

Performance is comparable or better than baseline models on deraining and desnowing.

Enables blind image restoration through a simple fusion scheme.

Effective feature sharing across tasks demonstrated by ablation studies.

Abstract

We present the Cross-stitched Multi-task Unified Dual Recursive Network (CMUDRN) model targeting the task of unified deraining and desnowing in a multi-task learning setting. This unified model borrows from the basic Dual Recursive Network (DRN) architecture developed by Cai et al. The proposed model makes use of cross-stitch units that enable multi-task learning across two separate DRN models, each tasked for single image deraining and desnowing, respectively. By fixing cross-stitch units at several layers of basic task-specific DRN networks, we perform multi-task learning over the two separate DRN models. To enable blind image restoration, on top of these structures we employ a simple neural fusion scheme which merges the output of each DRN. The separate task-specific DRN models and the fusion scheme are simultaneously trained by enforcing local and global supervision. Local supervision is applied on the two DRN submodules, and global supervision is applied on the data fusion submodule of the proposed model. Consequently, we both enable feature sharing across task-specific DRN models and control the image restoration behavior of the DRN submodules. An ablation study shows the strength of the hypothesized CMUDRN model, and experiments indicate that its performance is comparable or better than baseline DRN models on the single image deraining and desnowing tasks. Moreover, CMUDRN enables blind image restoration for the two underlying image restoration tasks, by unifying task-specific image restoration pipelines via a naive parametric fusion scheme. The CMUDRN implementation is available at https://github.com/VCL3D/CMUDRN.