Real-time Motion Segmentation with Event-based Normal Flow

TL;DR

This paper introduces a real-time motion segmentation method for event-based cameras using normal flow, significantly improving efficiency and accuracy over previous approaches by formulating the task as an energy minimization problem solved with graph cuts.

Contribution

The work presents a novel normal flow-based framework that efficiently estimates motion models for moving objects in event-based vision, enabling real-time performance with minimal candidate models.

Findings

Achieves nearly 800x speedup compared to state-of-the-art methods.

Demonstrates high accuracy on multiple public datasets.

Enables real-time motion segmentation in challenging scenarios.

Abstract

Event-based cameras are bio-inspired sensors with pixels that independently and asynchronously respond to brightness changes at microsecond resolution, offering the potential to handle visual tasks in challenging scenarios. However, due to the sparse information content in individual events, directly processing the raw event data to solve vision tasks is highly inefficient, which severely limits the applicability of state-of-the-art methods in real-time tasks, such as motion segmentation, a fundamental task for dynamic scene understanding. Incorporating normal flow as an intermediate representation to compress motion information from event clusters within a localized region provides a more effective solution. In this work, we propose a normal flow-based motion segmentation framework for event-based vision. Leveraging the dense normal flow directly learned from event neighborhoods as input, we formulate the motion segmentation task as an energy minimization problem solved via graph cuts, and optimize it iteratively with normal flow clustering and motion model fitting. By using a normal flow-based motion model initialization and fitting method, the proposed system is able to efficiently estimate the motion models of independently moving objects with only a limited number of candidate models, which significantly reduces the computational complexity and ensures real-time performance, achieving nearly a 800x speedup in comparison to the open-source state-of-the-art method. Extensive evaluations on multiple public datasets fully demonstrate the accuracy and efficiency of our framework.