A Multi-model Approach for Video Data Retrieval in Autonomous Vehicle Development

TL;DR

This paper introduces a multi-model pipeline that enables engineers to search vehicle logs using natural language descriptions, simplifying scenario retrieval in autonomous vehicle development.

Contribution

It proposes a novel multi-model architecture and interface for natural language-based retrieval of vehicle log scenarios, reducing reliance on complex SQL queries.

Findings

Achieved a mean evaluation score of 3.3 by engineers.

Demonstrated improved workflow for scenario search in vehicle logs.

Presented a visualization interface for query and results.

Abstract

Autonomous driving software generates enormous amounts of data every second, which software development organizations save for future analysis and testing in the form of logs. However, given the vast size of this data, locating specific scenarios within a collection of vehicle logs can be challenging. Writing the correct SQL queries to find these scenarios requires engineers to have a strong background in SQL and the specific databases in question, further complicating the search process. This paper presents and evaluates a pipeline that allows searching for specific scenarios in log collections using natural language descriptions instead of SQL. The generated descriptions were evaluated by engineers working with vehicle logs at the Zenseact on a scale from 1 to 5. Our approach achieved a mean score of 3.3, demonstrating the potential of using a multi-model architecture to improve the software development workflow. We also present an interface that can visualize the query process and visualize the results.