Spatio-Temporal State Space Model For Efficient Event-Based Optical Flow

TL;DR

This paper introduces a spatio-temporal state space model for event-based optical flow that significantly improves computational efficiency while maintaining competitive accuracy, enabling real-time applications.

Contribution

The authors propose a novel Spatio-Temporal State Space Model (STSSM) module and architecture that effectively captures spatio-temporal correlations in event data, outperforming existing methods in efficiency.

Findings

4.5x faster inference than TMA

8x lower computations than TMA

2x lower computations than EV-FlowNet

Abstract

Event cameras unlock new frontiers that were previously unthinkable with standard frame-based cameras. One notable example is low-latency motion estimation (optical flow), which is critical for many real-time applications. In such applications, the computational efficiency of algorithms is paramount. Although recent deep learning paradigms such as CNN, RNN, or ViT have shown remarkable performance, they often lack the desired computational efficiency. Conversely, asynchronous event-based methods including SNNs and GNNs are computationally efficient; however, these approaches fail to capture sufficient spatio-temporal information, a powerful feature required to achieve better performance for optical flow estimation. In this work, we introduce Spatio-Temporal State Space Model (STSSM) module along with a novel network architecture to develop an extremely efficient solution with competitive performance. Our STSSM module leverages state-space models to effectively capture spatio-temporal correlations in event data, offering higher performance with lower complexity compared to ViT, CNN-based architectures in similar settings. Our model achieves 4.5x faster inference and 8x lower computations compared to TMA and 2x lower computations compared to EV-FlowNet with competitive performance on the DSEC benchmark. Our code will be available at https://github.com/AhmedHumais/E-STMFlow