Vision-Language Feature Alignment for Road Anomaly Segmentation

TL;DR

This paper introduces VL-Anomaly, a novel vision-language framework for road anomaly segmentation that leverages pre-trained models and multi-source inference to improve safety-critical detection accuracy in autonomous systems.

Contribution

The paper proposes a new vision-language anomaly segmentation method using semantic priors and multi-source inference, significantly enhancing detection accuracy over existing pixel-based approaches.

Findings

Achieves state-of-the-art results on RoadAnomaly, SMIYC, and Fishyscapes datasets.

Effectively suppresses false positives in background regions.

Improves recall of true out-of-distribution anomalies.

Abstract

Safe autonomous systems in complex environments require robust road anomaly segmentation to identify unknown obstacles. However, existing approaches often rely on pixel-level statistics to determine whether a region appears anomalous. This reliance leads to high false-positive rates on semantically normal background regions such as sky or vegetation, and poor recall of true Out-of-distribution (OOD) instances, thereby posing safety risks for robotic perception and decision-making. To address these challenges, we propose VL-Anomaly, a vision-language anomaly segmentation framework that incorporates semantic priors from pre-trained Vision-Language Models (VLMs). Specifically, we design a prompt learning-driven alignment module that adapts Mask2Forme's visual features to CLIP text embeddings of known categories, effectively suppressing spurious anomaly responses in background regions. At inference time, we further introduce a multi-source inference strategy that integrates text-guided similarity, CLIP-based image-text similarity and detector confidence, enabling more reliable anomaly prediction by leveraging complementary information sources. Extensive experiments demonstrate that VL-Anomaly achieves state-of-the-art performance on benchmark datasets including RoadAnomaly, SMIYC and Fishyscapes.Code is released on https://github.com/NickHezhuolin/VL-aligner-Road-anomaly-segment.