Bilateral Event Mining and Complementary for Event Stream Super-Resolution

TL;DR

This paper introduces BMCNet, a novel two-stream network with bilateral information exchange for event stream super-resolution, significantly improving resolution and downstream task performance by effectively modeling positive and negative events separately.

Contribution

The paper proposes a bilateral event mining and complementary network (BMCNet) with a bilateral information exchange module to better model and utilize positive and negative events in ESR.

Findings

Outperforms previous state-of-the-art methods by over 11% in ESR accuracy.

Enhances downstream tasks like object recognition and video reconstruction.

Demonstrates effectiveness on both real and synthetic datasets.

Abstract

Event Stream Super-Resolution (ESR) aims to address the challenge of insufficient spatial resolution in event streams, which holds great significance for the application of event cameras in complex scenarios. Previous works for ESR often process positive and negative events in a mixed paradigm. This paradigm limits their ability to effectively model the unique characteristics of each event and mutually refine each other by considering their correlations. In this paper, we propose a bilateral event mining and complementary network (BMCNet) to fully leverage the potential of each event and capture the shared information to complement each other simultaneously. Specifically, we resort to a two-stream network to accomplish comprehensive mining of each type of events individually. To facilitate the exchange of information between two streams, we propose a bilateral information exchange (BIE) module. This module is layer-wisely embedded between two streams, enabling the effective propagation of hierarchical global information while alleviating the impact of invalid information brought by inherent characteristics of events. The experimental results demonstrate that our approach outperforms the previous state-of-the-art methods in ESR, achieving performance improvements of over 11\% on both real and synthetic datasets. Moreover, our method significantly enhances the performance of event-based downstream tasks such as object recognition and video reconstruction. Our code is available at https://github.com/Lqm26/BMCNet-ESR.