Nonlinear Motion-Guided and Spatio-Temporal Aware Network for Unsupervised Event-Based Optical Flow

TL;DR

This paper introduces E-NMSTFlow, an unsupervised neural network that leverages spatio-temporal features and nonlinear motion compensation to improve event-based optical flow estimation, especially over long sequences.

Contribution

It proposes novel modules and a nonlinear motion loss that effectively utilize rich spatio-temporal information and nonlinear motion modeling for better optical flow accuracy.

Findings

Ranks first among unsupervised methods on MVSEC and DSEC-Flow datasets.

Demonstrates significant improvements over frame-based techniques.

Effectively captures nonlinear motion in long sequences.

Abstract

Event cameras have the potential to capture continuous motion information over time and space, making them well-suited for optical flow estimation. However, most existing learning-based methods for event-based optical flow adopt frame-based techniques, ignoring the spatio-temporal characteristics of events. Additionally, these methods assume linear motion between consecutive events within the loss time window, which increases optical flow errors in long-time sequences. In this work, we observe that rich spatio-temporal information and accurate nonlinear motion between events are crucial for event-based optical flow estimation. Therefore, we propose E-NMSTFlow, a novel unsupervised event-based optical flow network focusing on long-time sequences. We propose a Spatio-Temporal Motion Feature Aware (STMFA) module and an Adaptive Motion Feature Enhancement (AMFE) module, both of which utilize rich spatio-temporal information to learn spatio-temporal data associations. Meanwhile, we propose a nonlinear motion compensation loss that utilizes the accurate nonlinear motion between events to improve the unsupervised learning of our network. Extensive experiments demonstrate the effectiveness and superiority of our method. Remarkably, our method ranks first among unsupervised learning methods on the MVSEC and DSEC-Flow datasets. Our project page is available at https://wynelio.github.io/E-NMSTFlow.