Turb-Seg-Res: A Segment-then-Restore Pipeline for Dynamic Videos with Atmospheric Turbulence

TL;DR

This paper introduces a novel segment-then-restore pipeline for dynamic videos affected by atmospheric turbulence, utilizing motion segmentation, turbulence modeling, and a transformer-based restoration approach to improve video clarity.

Contribution

It presents the first pipeline specifically designed for restoring dynamic scene videos in turbulent environments, combining motion segmentation, turbulence simulation, and transformer-based restoration.

Findings

Effective removal of geometric distortion

Significant enhancement of video sharpness

Outperforms existing restoration methods

Abstract

Tackling image degradation due to atmospheric turbulence, particularly in dynamic environment, remains a challenge for long-range imaging systems. Existing techniques have been primarily designed for static scenes or scenes with small motion. This paper presents the first segment-then-restore pipeline for restoring the videos of dynamic scenes in turbulent environment. We leverage mean optical flow with an unsupervised motion segmentation method to separate dynamic and static scene components prior to restoration. After camera shake compensation and segmentation, we introduce foreground/background enhancement leveraging the statistics of turbulence strength and a transformer model trained on a novel noise-based procedural turbulence generator for fast dataset augmentation. Benchmarked against existing restoration methods, our approach restores most of the geometric distortion and enhances sharpness for videos. We make our code, simulator, and data publicly available to advance the field of video restoration from turbulence: riponcs.github.io/TurbSegRes