Modulo Video Recovery via Selective Spatiotemporal Vision Transformer

TL;DR

This paper introduces SSViT, a novel deep learning framework using a selective spatiotemporal transformer for high-quality modulo video reconstruction, outperforming previous methods in efficiency and accuracy.

Contribution

We develop the first deep learning-based modulo video recovery method employing a selective spatiotemporal transformer architecture.

Findings

SSViT achieves state-of-the-art performance in modulo video reconstruction.

The token selection strategy improves computational efficiency and focus.

High-quality reconstructions are possible from 8-bit folded videos.

Abstract

Conventional image sensors have limited dynamic range, causing saturation in high-dynamic-range (HDR) scenes. Modulo cameras address this by folding incident irradiance into a bounded range, yet require specialized unwrapping algorithms to reconstruct the underlying signal. Unlike HDR recovery, which extends dynamic range from conventional sampling, modulo recovery restores actual values from folded samples. Despite being introduced over a decade ago, progress in modulo image recovery has been slow, especially in the use of modern deep learning techniques. In this work, we demonstrate that standard HDR methods are unsuitable for modulo recovery. Transformers, however, can capture global dependencies and spatial-temporal relationships crucial for resolving folded video frames. Still, adapting existing Transformer architectures for modulo recovery demands novel techniques. To this end, we present Selective Spatiotemporal Vision Transformer (SSViT), the first deep learning framework for modulo video reconstruction. SSViT employs a token selection strategy to improve efficiency and concentrate on the most critical regions. Experiments confirm that SSViT produces high-quality reconstructions from 8-bit folded videos and achieves state-of-the-art performance in modulo video recovery.