Video Compressive Sensing for Spatial Multiplexing Cameras using Motion-Flow Models

TL;DR

This paper introduces CS-MUVI, a novel compressive sensing framework for spatial multiplexing cameras that enables high-quality video recovery at high compression ratios by leveraging multi-scale sensing and optical flow constraints.

Contribution

The paper proposes a new sensing and recovery framework that combines multi-scale sensing matrices with optical flow constraints for improved video reconstruction in SMCs.

Findings

High-quality video can be recovered at roughly 60x compression.

The framework enables efficient low-resolution previews and high-resolution recovery.

Optical flow constraints improve reconstruction quality.

Abstract

Spatial multiplexing cameras (SMCs) acquire a (typically static) scene through a series of coded projections using a spatial light modulator (e.g., a digital micro-mirror device) and a few optical sensors. This approach finds use in imaging applications where full-frame sensors are either too expensive (e.g., for short-wave infrared wavelengths) or unavailable. Existing SMC systems reconstruct static scenes using techniques from compressive sensing (CS). For videos, however, existing acquisition and recovery methods deliver poor quality. In this paper, we propose the CS multi-scale video (CS-MUVI) sensing and recovery framework for high-quality video acquisition and recovery using SMCs. Our framework features novel sensing matrices that enable the efficient computation of a low-resolution video preview, while enabling high-resolution video recovery using convex optimization. To further improve the quality of the reconstructed videos, we extract optical-flow estimates from the low-resolution previews and impose them as constraints in the recovery procedure. We demonstrate the efficacy of our CS-MUVI framework for a host of synthetic and real measured SMC video data, and we show that high-quality videos can be recovered at roughly $60\times$ compression.