Railway Artificial Intelligence Learning Benchmark (RAIL-BENCH): A Benchmark Suite for Perception in the Railway Domain

TL;DR

RAIL-BENCH is the first comprehensive benchmark suite for perception tasks in railway environments, enabling standardized evaluation and comparison of AI approaches for automated train operation.

Contribution

It introduces RAIL-BENCH, a novel benchmark suite with diverse datasets, evaluation metrics, and a new line detection metric, addressing the lack of standardized tools in railway perception research.

Findings

Provides curated datasets from real-world railway scenarios.

Introduces LineAP, a new metric for line detection accuracy.

Includes public scoreboards for benchmarking progress.

Abstract

Automated train operation on existing railway infrastructure requires robust camera-based perception, yet the railway domain lacks public benchmark suites with standardized evaluation protocols that would enable reproducible comparison of approaches. We present RAIL-BENCH, the first perception benchmark suite for the railway domain. It comprises five challenges - rail track detection, object detection, vegetation segmentation, multi-object tracking, and monocular visual odometry - each tailored to the specific characteristics of railway environments. RAIL-BENCH provides curated training and test datasets drawn from diverse real-world scenarios, evaluation metrics, and public scoreboards (https://www.mrt.kit.edu/railbench). For the rail track detection challenge we introduce LineAP, a novel segment-based average precision metric that evaluates the geometric accuracy of polyline predictions independently of instance-level grouping, addressing key limitations of existing line detection metrics.