SBNet: Segmentation-based Network for Natural Language-based Vehicle Search

TL;DR

SBNet is a deep neural network designed for natural language-based vehicle retrieval, utilizing segmentation and specialized modules to handle multi-modal data and improve accuracy in vehicle search tasks.

Contribution

Introduces SBNet, a novel segmentation-based neural network with task-specific modules for enhanced natural language vehicle retrieval performance.

Findings

Significant improvement over baseline in AI City Challenge 2021

Effective handling of multi-modal data with substitution and future prediction modules

Trained on CityFlow-NL dataset with 2,498 vehicle tracks and descriptions

Abstract

Natural language-based vehicle retrieval is a task to find a target vehicle within a given image based on a natural language description as a query. This technology can be applied to various areas including police searching for a suspect vehicle. However, it is challenging due to the ambiguity of language descriptions and the difficulty of processing multi-modal data. To tackle this problem, we propose a deep neural network called SBNet that performs natural language-based segmentation for vehicle retrieval. We also propose two task-specific modules to improve performance: a substitution module that helps features from different domains to be embedded in the same space and a future prediction module that learns temporal information. SBnet has been trained using the CityFlow-NL dataset that contains 2,498 tracks of vehicles with three unique natural language descriptions each and tested 530 unique vehicle tracks and their corresponding query sets. SBNet achieved a significant improvement over the baseline in the natural language-based vehicle tracking track in the AI City Challenge 2021.