ELMformer: Efficient Raw Image Restoration with a Locally Multiplicative Transformer

TL;DR

ELMformer is a novel transformer-based model designed for efficient raw image restoration, utilizing a bi-directional fusion module and locally multiplicative self-attention to outperform existing methods in quality and computational efficiency.

Contribution

The paper introduces ELMformer, a new raw image restoration model with two innovative modules that enhance performance and reduce computational costs compared to prior approaches.

Findings

Achieves highest performance and lowest FLOPs on raw denoising and deblurring benchmarks.

Outperforms ISP-based methods on SIDD benchmark without needing additional sRGB images.

Demonstrates superior generalization and efficiency in raw image restoration tasks.

Abstract

In order to get raw images of high quality for downstream Image Signal Process (ISP), in this paper we present an Efficient Locally Multiplicative Transformer called ELMformer for raw image restoration. ELMformer contains two core designs especially for raw images whose primitive attribute is single-channel. The first design is a Bi-directional Fusion Projection (BFP) module, where we consider both the color characteristics of raw images and spatial structure of single-channel. The second one is that we propose a Locally Multiplicative Self-Attention (L-MSA) scheme to effectively deliver information from the local space to relevant parts. ELMformer can efficiently reduce the computational consumption and perform well on raw image restoration tasks. Enhanced by these two core designs, ELMformer achieves the highest performance and keeps the lowest FLOPs on raw denoising and raw deblurring benchmarks compared with state-of-the-arts. Extensive experiments demonstrate the superiority and generalization ability of ELMformer. On SIDD benchmark, our method has even better denoising performance than ISP-based methods which need huge amount of additional sRGB training images. The codes are release at https://github.com/leonmakise/ELMformer.