Depth-Enhanced YOLO-SAM2 Detection for Reliable Ballast Insufficiency Identification

TL;DR

This paper introduces a depth-enhanced YOLO-SAM2 framework that significantly improves the detection of ballast insufficiency in railway tracks by integrating depth correction and segmentation, leading to higher recall and F1-scores.

Contribution

The novel integration of depth correction and SAM2 segmentation with YOLO-SAM2 enhances ballast insufficiency detection accuracy and reliability in railway inspections.

Findings

Recall improved from 0.49 to 0.80 with depth enhancement.

F1-score increased from 0.66 to over 0.80.

Depth correction significantly boosts safety-critical detection performance.

Abstract

This paper presents a depth-enhanced YOLO-SAM2 framework for detecting ballast insufficiency in railway tracks using RGB-D data. Although YOLOv8 provides reliable localization, the RGB-only model shows limited safety performance, achieving high precision (0.99) but low recall (0.49) due to insufficient ballast, as it tends to over-predict the sufficient class. To improve reliability, we incorporate depth-based geometric analysis enabled by a sleeper-aligned depth-correction pipeline that compensates for RealSense spatial distortion using polynomial modeling, RANSAC, and temporal smoothing. SAM2 segmentation further refines region-of-interest masks, enabling accurate extraction of sleeper and ballast profiles for geometric classification. Experiments on field-collected top-down RGB-D data show that depth-enhanced configurations substantially improve the detection of insufficient ballast. Depending on bounding-box sampling (AABB or RBB) and geometric criteria, recall increases from 0.49 to as high as 0.80, and F1-score improves from 0.66 to over 0.80. These results demonstrate that integrating depth correction with YOLO-SAM2 yields a more robust and reliable approach for automated railway ballast inspection, particularly in visually ambiguous or safety-critical scenarios.