REALM: An RGB and Event Aligned Latent Manifold for Cross-Modal Perception

TL;DR

REALM is a novel cross-modal framework that aligns event camera data with RGB foundation models, enabling zero-shot downstream tasks and state-of-the-art feature matching without task-specific training.

Contribution

It introduces a method to project event data into RGB model latent space using LoRA, allowing generalization and zero-shot application of image decoders to event streams.

Findings

Effective mapping of events into ViT-based latent space.

Enables zero-shot depth estimation and semantic segmentation.

Achieves state-of-the-art performance in wide-baseline feature matching.

Abstract

Event cameras provide several unique advantages over standard frame-based sensors, including high temporal resolution, low latency, and robustness to extreme lighting. However, existing learning-based approaches for event processing are typically confined to narrow, task-specific silos and lack the ability to generalize across modalities. We address this gap with REALM, a cross-modal framework that learns an RGB and Event Aligned Latent Manifold by projecting event representations into the pretrained latent space of RGB foundation models. Instead of task-specific training, we leverage low-rank adaptation (LoRA) to bridge the modality gap, effectively unlocking the geometric and semantic priors of frozen RGB backbones for asynchronous event streams. We demonstrate that REALM effectively maps events into the ViT-based foundation latent space. Our method allows us to perform downstream tasks like depth estimation and semantic segmentation by simply transferring linear heads trained on the RGB teacher. Most significantly, REALM enables the direct, zero-shot application of complex, frozen image-trained decoders, such as MASt3R, to raw event data. We demonstrate state-of-the-art performance in wide-baseline feature matching, significantly outperforming specialized architectures. Code and models are available upon acceptance.