Motion-aware Dynamic Graph Neural Network for Video Compressive Sensing

TL;DR

This paper introduces a motion-aware dynamic graph neural network that effectively captures long-range spatial and temporal dependencies for improved video compressive sensing reconstruction, outperforming existing methods in accuracy and efficiency.

Contribution

It proposes a novel motion-aware dynamic GNN that models non-local pixel interactions in space and time, enhancing video SCI reconstruction capabilities.

Findings

Demonstrates superior reconstruction quality on simulation and real data

Shows improved efficiency over existing methods

Visualizations reveal effective dynamic sampling operations

Abstract

Video snapshot compressive imaging (SCI) utilizes a 2D detector to capture sequential video frames and compress them into a single measurement. Various reconstruction methods have been developed to recover the high-speed video frames from the snapshot measurement. However, most existing reconstruction methods are incapable of efficiently capturing long-range spatial and temporal dependencies, which are critical for video processing. In this paper, we propose a flexible and robust approach based on the graph neural network (GNN) to efficiently model non-local interactions between pixels in space and time regardless of the distance. Specifically, we develop a motion-aware dynamic GNN for better video representation, i.e., represent each node as the aggregation of relative neighbors under the guidance of frame-by-frame motions, which consists of motion-aware dynamic sampling, cross-scale node sampling, global knowledge integration, and graph aggregation. Extensive results on both simulation and real data demonstrate both the effectiveness and efficiency of the proposed approach, and the visualization illustrates the intrinsic dynamic sampling operations of our proposed model for boosting the video SCI reconstruction results. The code and model will be released.