DeTurb: Atmospheric Turbulence Mitigation with Deformable 3D Convolutions and 3D Swin Transformers

TL;DR

DeTurb introduces a novel deep learning framework combining deformable 3D convolutions and 3D Swin Transformers to effectively mitigate atmospheric turbulence effects in long-range imaging, improving image quality with reasonable computational efficiency.

Contribution

The paper presents a new deep learning approach that integrates geometric restoration with enhancement modules using deformable 3D convolutions and 3D Swin Transformers, addressing spatiotemporal turbulence distortions effectively.

Findings

Outperforms state-of-the-art methods on synthetic turbulence data

Achieves superior image restoration quality on real turbulence effects

Maintains reasonable speed and model size for practical applications

Abstract

Atmospheric turbulence in long-range imaging significantly degrades the quality and fidelity of captured scenes due to random variations in both spatial and temporal dimensions. These distortions present a formidable challenge across various applications, from surveillance to astronomy, necessitating robust mitigation strategies. While model-based approaches achieve good results, they are very slow. Deep learning approaches show promise in image and video restoration but have struggled to address these spatiotemporal variant distortions effectively. This paper proposes a new framework that combines geometric restoration with an enhancement module. Random perturbations and geometric distortion are removed using a pyramid architecture with deformable 3D convolutions, resulting in aligned frames. These frames are then used to reconstruct a sharp, clear image via a multi-scale architecture of 3D Swin Transformers. The proposed framework demonstrates superior performance over the state of the art for both synthetic and real atmospheric turbulence effects, with reasonable speed and model size.