Rethinking Transformer-Based Blind-Spot Network for Self-Supervised Image Denoising

TL;DR

This paper introduces a Transformer-based blind-spot network for self-supervised image denoising, redesigning attention mechanisms to meet blind-spot constraints and improve denoising performance.

Contribution

It proposes a novel TBSN architecture with redesigned channel and spatial attentions that satisfy blind-spot requirements, enhancing denoising capabilities.

Findings

TBSN extends the receptive field significantly.

TBSN outperforms state-of-the-art SSID methods.

Knowledge distillation improves efficiency without performance loss.

Abstract

Blind-spot networks (BSN) have been prevalent neural architectures in self-supervised image denoising (SSID). However, most existing BSNs are conducted with convolution layers. Although transformers have shown the potential to overcome the limitations of convolutions in many image restoration tasks, the attention mechanisms may violate the blind-spot requirement, thereby restricting their applicability in BSN. To this end, we propose to analyze and redesign the channel and spatial attentions to meet the blind-spot requirement. Specifically, channel self-attention may leak the blind-spot information in multi-scale architectures, since the downsampling shuffles the spatial feature into channel dimensions. To alleviate this problem, we divide the channel into several groups and perform channel attention separately. For spatial selfattention, we apply an elaborate mask to the attention matrix to restrict and mimic the receptive field of dilated convolution. Based on the redesigned channel and window attentions, we build a Transformer-based Blind-Spot Network (TBSN), which shows strong local fitting and global perspective abilities. Furthermore, we introduce a knowledge distillation strategy that distills TBSN into smaller denoisers to improve computational efficiency while maintaining performance. Extensive experiments on real-world image denoising datasets show that TBSN largely extends the receptive field and exhibits favorable performance against state-of-theart SSID methods.