RGB-Event HyperGraph Prompt for Kilometer Marker Recognition based on Pre-trained Foundation Models

TL;DR

This paper introduces a novel RGB-Event hypergraph prompt method for kilometer marker recognition in metro environments, utilizing pre-trained foundation models and a new large-scale dataset to improve localization accuracy under challenging conditions.

Contribution

It presents a multi-modal adaptation of a pre-trained OCR model for kilometer marker recognition and introduces the first large-scale RGB-Event dataset for this task.

Findings

Effective recognition in complex metro environments

Superior performance on EvMetro5K and other benchmarks

First large-scale RGB-Event dataset for KMR

Abstract

Metro trains often operate in highly complex environments, characterized by illumination variations, high-speed motion, and adverse weather conditions. These factors pose significant challenges for visual perception systems, especially those relying solely on conventional RGB cameras. To tackle these difficulties, we explore the integration of event cameras into the perception system, leveraging their advantages in low-light conditions, high-speed scenarios, and low power consumption. Specifically, we focus on Kilometer Marker Recognition (KMR), a critical task for autonomous metro localization under GNSS-denied conditions. In this context, we propose a robust baseline method based on a pre-trained RGB OCR foundation model, enhanced through multi-modal adaptation. Furthermore, we construct the first large-scale RGB-Event dataset, EvMetro5K, containing 5,599 pairs of synchronized RGB-Event samples, split into 4,479 training and 1,120 testing samples. Extensive experiments on EvMetro5K and other widely used benchmarks demonstrate the effectiveness of our approach for KMR. Both the dataset and source code will be released on https://github.com/Event-AHU/EvMetro5K_benchmark