Segment Any Vehicle: Semantic and Visual Context Driven SAM and A Benchmark

TL;DR

This paper introduces SAV, a novel vehicle part segmentation framework that combines a SAM-based encoder-decoder, a vehicle part knowledge graph, and a context retrieval module, along with a new large-scale dataset VehicleSeg10K.

Contribution

It presents a new framework for vehicle part segmentation that overcomes SAM's limitations and introduces a comprehensive benchmark dataset for the task.

Findings

SAV outperforms baseline models on VehicleSeg10K and other datasets.

The knowledge graph improves semantic understanding of vehicle parts.

Context retrieval enhances segmentation accuracy in diverse scenes.

Abstract

With the rapid advancement of autonomous driving, vehicle perception, particularly detection and segmentation, has placed increasingly higher demands on algorithmic performance. Pre-trained large segmentation models, especially Segment Anything Model (SAM), have sparked significant interest and inspired new research directions in artificial intelligence. However, SAM cannot be directly applied to the fine-grained task of vehicle part segmentation, as its text-prompted segmentation functionality is not publicly accessible, and the mask regions generated by its default mode lack semantic labels, limiting its utility in structured, category-specific segmentation tasks. To address these limitations, we propose SAV, a novel framework comprising three core components: a SAM-based encoder-decoder, a vehicle part knowledge graph, and a context sample retrieval encoding module. The knowledge graph explicitly models the spatial and geometric relationships among vehicle parts through a structured ontology, effectively encoding prior structural knowledge. Meanwhile, the context retrieval module enhances segmentation by identifying and leveraging visually similar vehicle instances from training data, providing rich contextual priors for improved generalization. Furthermore, we introduce a new large-scale benchmark dataset for vehicle part segmentation, named VehicleSeg10K, which contains 11,665 high-quality pixel-level annotations across diverse scenes and viewpoints. We conduct comprehensive experiments on this dataset and two other datasets, benchmarking multiple representative baselines to establish a solid foundation for future research and comparison. % Both the dataset and source code of this paper will be released upon acceptance. Both the dataset and source code of this paper will be released on https://github.com/Event-AHU/SAV