Optical Flow Estimation for Spiking Camera

TL;DR

This paper introduces SCFlow, a deep learning pipeline for estimating optical flow from spike streams of high-speed scenes captured by bio-inspired spiking cameras, addressing data modality challenges.

Contribution

The paper proposes a novel input representation and synthesizes specialized datasets for training, enabling effective optical flow estimation from spike streams.

Findings

SCFlow accurately predicts optical flow in high-speed scenes.

The method generalizes well to real spike streams.

The approach outperforms existing methods on synthetic datasets.

Abstract

As a bio-inspired sensor with high temporal resolution, the spiking camera has an enormous potential in real applications, especially for motion estimation in high-speed scenes. However, frame-based and event-based methods are not well suited to spike streams from the spiking camera due to the different data modalities. To this end, we present, SCFlow, a tailored deep learning pipeline to estimate optical flow in high-speed scenes from spike streams. Importantly, a novel input representation is introduced which can adaptively remove the motion blur in spike streams according to the prior motion. Further, for training SCFlow, we synthesize two sets of optical flow data for the spiking camera, SPIkingly Flying Things and Photo-realistic High-speed Motion, denoted as SPIFT and PHM respectively, corresponding to random high-speed and well-designed scenes. Experimental results show that the SCFlow can predict optical flow from spike streams in different high-speed scenes. Moreover, SCFlow shows promising generalization on \textbf{real spike streams}. Codes and datasets refer to https://github.com/Acnext/Optical-Flow-For-Spiking-Camera.