An Empirical study on LLM-based Log Retrieval for Software Engineering Metadata Management

TL;DR

This paper presents an LLM-supported method for natural language log retrieval in autonomous driving systems, combining signal logs and videos to improve efficiency and reliability over traditional SQL queries.

Contribution

It introduces a novel LLM-based approach that integrates log data and videos for scenario search, with quantifiable metrics and an API implementation, enhancing retrieval accuracy and usability.

Findings

Improved efficiency in scenario retrieval

Enhanced reliability without relying solely on SQL

Effective integration of logs and videos for visualization

Abstract

Developing autonomous driving systems (ADSs) involves generating and storing extensive log data from test drives, which is essential for verification, research, and simulation. However, these high-frequency logs, recorded over varying durations, pose challenges for developers attempting to locate specific driving scenarios. This difficulty arises due to the wide range of signals representing various vehicle components and driving conditions, as well as unfamiliarity of some developers' with the detailed meaning of these signals. Traditional SQL-based querying exacerbates this challenge by demanding both domain expertise and database knowledge, often yielding results that are difficult to verify for accuracy. This paper introduces a Large Language Model (LLM)-supported approach that combines signal log data with video recordings from test drives, enabling natural language based scenario searches while reducing the need for specialized knowledge. By leveraging scenario distance graphs and relative gap indicators, it provides quantifiable metrics to evaluate the reliability of query results. The method is implemented as an API for efficient database querying and retrieval of relevant records, paired with video frames for intuitive visualization. Evaluation on an open industrial dataset demonstrates improved efficiency and reliability in scenario retrieval, eliminating dependency on a single data source and conventional SQL.