Decomposer: Semi-supervised Learning of Image Restoration and Image Decomposition

TL;DR

Decomposer is a semi-supervised transformer-based model that decomposes distorted image sequences into original images and their augmentations, improving understanding of complex image distortions using the SIDAR dataset.

Contribution

The paper introduces a novel semi-supervised transformer model for decomposing distorted images into fundamental components, leveraging weak supervision and the SIDAR dataset.

Findings

Effective separation of image components achieved

Pre-training on pseudo labels enhances decomposition accuracy

Model handles various distortions like shadows, lighting, and occlusions

Abstract

We present Decomposer, a semi-supervised reconstruction model that decomposes distorted image sequences into their fundamental building blocks - the original image and the applied augmentations, i.e., shadow, light, and occlusions. To solve this problem, we use the SIDAR dataset that provides a large number of distorted image sequences: each sequence contains images with shadows, lighting, and occlusions applied to an undistorted version. Each distortion changes the original signal in different ways, e.g., additive or multiplicative noise. We propose a transformer-based model to explicitly learn this decomposition. The sequential model uses 3D Swin-Transformers for spatio-temporal encoding and 3D U-Nets as prediction heads for individual parts of the decomposition. We demonstrate that by separately pre-training our model on weakly supervised pseudo labels, we can steer our model to optimize for our ambiguous problem definition and learn to differentiate between the different image distortions.