Fusing Frame and Event Vision for High-speed Optical Flow for Edge Application

TL;DR

This paper introduces a bio-mimetic fusion of frame and event-based vision to achieve high-speed optical flow with low error, suitable for real-time drone applications.

Contribution

It presents a novel fusion approach combining frame and event data, enabling high-speed, low-error optical flow computation for edge applications.

Findings

19% error degradation at 4x speed up on MVSEC dataset

System enables real-time high-speed drone tracking and segmentation

Fusion overcomes fundamental trade-offs in frame-based processing

Abstract

Optical flow computation with frame-based cameras provides high accuracy but the speed is limited either by the model size of the algorithm or by the frame rate of the camera. This makes it inadequate for high-speed applications. Event cameras provide continuous asynchronous event streams overcoming the frame-rate limitation. However, the algorithms for processing the data either borrow frame like setup limiting the speed or suffer from lower accuracy. We fuse the complementary accuracy and speed advantages of the frame and event-based pipelines to provide high-speed optical flow while maintaining a low error rate. Our bio-mimetic network is validated with the MVSEC dataset showing 19% error degradation at 4x speed up. We then demonstrate the system with a high-speed drone flight scenario where a high-speed event camera computes the flow even before the optical camera sees the drone making it suited for applications like tracking and segmentation. This work shows the fundamental trade-offs in frame-based processing may be overcome by fusing data from other modalities.