Event-based Monocular Dense Depth Estimation with Recurrent Transformers

TL;DR

This paper introduces EReFormer, a novel event-based monocular depth estimation method using recurrent transformers that effectively exploits spatial and temporal information from asynchronous event streams, outperforming existing methods.

Contribution

The paper presents the first pure transformer-based recursive model for event-based monocular depth estimation, combining spatial and temporal modeling with improved efficiency.

Findings

EReFormer outperforms state-of-the-art methods on synthetic and real datasets.

The recursive transformer mechanism enhances temporal modeling capabilities.

The spatial transformer fusion module improves global context understanding.

Abstract

Event cameras, offering high temporal resolutions and high dynamic ranges, have brought a new perspective to address common challenges (e.g., motion blur and low light) in monocular depth estimation. However, how to effectively exploit the sparse spatial information and rich temporal cues from asynchronous events remains a challenging endeavor. To this end, we propose a novel event-based monocular depth estimator with recurrent transformers, namely EReFormer, which is the first pure transformer with a recursive mechanism to process continuous event streams. Technically, for spatial modeling, a novel transformer-based encoder-decoder with a spatial transformer fusion module is presented, having better global context information modeling capabilities than CNN-based methods. For temporal modeling, we design a gate recurrent vision transformer unit that introduces a recursive mechanism into transformers, improving temporal modeling capabilities while alleviating the expensive GPU memory cost. The experimental results show that our EReFormer outperforms state-of-the-art methods by a margin on both synthetic and real-world datasets. We hope that our work will attract further research to develop stunning transformers in the event-based vision community. Our open-source code can be found in the supplemental material.