EMatch: A Unified Framework for Event-based Optical Flow and Stereo Matching

TL;DR

This paper introduces EMatch, a unified deep learning framework that simultaneously performs event-based optical flow estimation and stereo matching by leveraging shared features and multi-task learning, achieving state-of-the-art results.

Contribution

It reformulates event-based optical flow and stereo matching as a single dense correspondence problem, enabling joint learning and cross-task transfer within one model.

Findings

Achieves state-of-the-art performance on both tasks.

Supports multi-task fusion and cross-task transfer without retraining.

Effectively utilizes shared features for both optical flow and stereo matching.

Abstract

Event cameras have shown promise in vision applications like optical flow estimation and stereo matching, with many specialized architectures leveraging the asynchronous and sparse nature of event data. However, existing works only focus event data within the confines of task-specific domains, overlooking how tasks across the temporal and spatial domains can reinforce each other. In this paper, we reformulate event-based flow estimation and stereo matching as a unified dense correspondence matching problem, enabling us to solve both tasks within a single model by directly matching features in a shared representation space. Specifically, our method utilizes a Temporal Recurrent Network to aggregate event features across temporal or spatial domains, and a Spatial Contextual Attention to enhance knowledge transfer across event flows via temporal or spatial interactions. By utilizing a shared feature similarities module that integrates knowledge from event streams via temporal or spatial interactions, our network performs optical flow estimation from temporal event segment inputs and stereo matching from spatial event segment inputs simultaneously. We demonstrate that our unified model inherently supports multi-task fusion and cross-task transfer. Without the need for retraining for specific task, our model can effectively handle both optical flow and stereo estimation, achieving state-of-the-art performance on both tasks.