MemNet: A Persistent Memory Network for Image Restoration

TL;DR

MemNet introduces a persistent memory mechanism in deep CNNs to effectively capture long-term dependencies, significantly improving performance across image denoising, super-resolution, and JPEG deblocking tasks.

Contribution

The paper proposes a novel deep persistent memory network with memory blocks that explicitly model long-term dependencies for image restoration.

Findings

MemNet outperforms state-of-the-art methods on all three image restoration tasks.

The memory mechanism effectively captures long-term dependencies in deep networks.

Experimental results demonstrate the necessity and superiority of MemNet.

Abstract

Recently, very deep convolutional neural networks (CNNs) have been attracting considerable attention in image restoration. However, as the depth grows, the long-term dependency problem is rarely realized for these very deep models, which results in the prior states/layers having little influence on the subsequent ones. Motivated by the fact that human thoughts have persistency, we propose a very deep persistent memory network (MemNet) that introduces a memory block, consisting of a recursive unit and a gate unit, to explicitly mine persistent memory through an adaptive learning process. The recursive unit learns multi-level representations of the current state under different receptive fields. The representations and the outputs from the previous memory blocks are concatenated and sent to the gate unit, which adaptively controls how much of the previous states should be reserved, and decides how much of the current state should be stored. We apply MemNet to three image restoration tasks, i.e., image denosing, super-resolution and JPEG deblocking. Comprehensive experiments demonstrate the necessity of the MemNet and its unanimous superiority on all three tasks over the state of the arts. Code is available at https://github.com/tyshiwo/MemNet.