Learning Flow-Guided Registration for RGB-Event Semantic Segmentation

TL;DR

This paper introduces BRENet, a flow-guided registration framework that improves RGB-Event semantic segmentation by aligning modalities through optical flow and a novel event tensor representation, addressing intrinsic misalignments.

Contribution

It recasts RGB-Event segmentation as a registration problem, proposing a novel flow-guided bidirectional framework and a dense event tensor representation, advancing the state-of-the-art in modality alignment.

Findings

Outperforms existing methods on four large-scale datasets.

Effectively bridges modality gaps with flow-guided registration.

Demonstrates robustness to motion lag and misalignment.

Abstract

Event cameras capture microsecond-level motion cues that complement RGB sensors. However, the prevailing paradigm of treating RGB-Event perception as a fusion problem is ill-posed, as it ignores the intrinsic (i) Spatiotemporal and (ii) Modal Misalignment, unlike other RGB-X sensing domains. To tackle these limitations, we recast RGB-Event segmentation from fusion to registration. We propose BRENet, a novel flow-guided bidirectional framework that adaptively matches correspondence between the asymmetric modalities. Specifically, it leverages temporally aligned optical flows as a coarse-grained guide, along with fine-grained event temporal features, to generate precise forward and backward pixel pairings for registration. This pairing mechanism converts the inherent motion lag into terms governed by flow estimation error, bridging modality gaps. Moreover, we introduce Motion-Enhanced Event Tensor (MET), a new representation that transforms sparse event streams into a dense, temporally coherent form. Extensive experiments on four large-scale datasets validate our approach, establishing flow-guided registration as a promising direction for RGB-Event segmentation. Our code is available at: https://github.com/zyaocoder/BRENet.