Hierarchical Separable Video Transformer for Snapshot Compressive Imaging

TL;DR

This paper introduces HiSViT, a hierarchical separable video transformer that improves snapshot compressive imaging reconstruction by focusing on multi-scale spatial interactions and reducing computational costs, outperforming previous methods.

Contribution

The paper proposes a novel hierarchical separable transformer architecture with CSS-MSA and GSM-FFN for efficient, multi-scale video reconstruction in SCI, addressing previous limitations in degradation insight.

Findings

Outperforms previous methods by >0.5 dB in PSNR.

Uses fewer parameters and lower complexity.

Effective multi-scale spatial-temporal modeling.

Abstract

Transformers have achieved the state-of-the-art performance on solving the inverse problem of Snapshot Compressive Imaging (SCI) for video, whose ill-posedness is rooted in the mixed degradation of spatial masking and temporal aliasing. However, previous Transformers lack an insight into the degradation and thus have limited performance and efficiency. In this work, we tailor an efficient reconstruction architecture without temporal aggregation in early layers and Hierarchical Separable Video Transformer (HiSViT) as building block. HiSViT is built by multiple groups of Cross-Scale Separable Multi-head Self-Attention (CSS-MSA) and Gated Self-Modulated Feed-Forward Network (GSM-FFN) with dense connections, each of which is conducted within a separate channel portions at a different scale, for multi-scale interactions and long-range modeling. By separating spatial operations from temporal ones, CSS-MSA introduces an inductive bias of paying more attention within frames instead of between frames while saving computational overheads. GSM-FFN further enhances the locality via gated mechanism and factorized spatial-temporal convolutions. Extensive experiments demonstrate that our method outperforms previous methods by $\!>\!0.5$ dB with comparable or fewer parameters and complexity. The source codes and pretrained models are released at https://github.com/pwangcs/HiSViT.