Have We Ever Encountered This Before? Retrieving Out-of-Distribution Road Obstacles from Driving Scenes

TL;DR

This paper introduces a novel method for retrieving out-of-distribution road obstacles from driving videos using text queries, aiding rapid testing and reconfiguration of autonomous driving systems in dynamic environments.

Contribution

It presents the first approach for text-based OoD object retrieval in driving scenes, combining OoD segmentation and multi-modal models for efficient scene extraction.

Findings

Effective retrieval of OoD obstacles from unlabeled videos

Demonstrated capability to curate OoD datasets for analysis

Enhanced tools for autonomous vehicle safety assessment

Abstract

In the life cycle of highly automated systems operating in an open and dynamic environment, the ability to adjust to emerging challenges is crucial. For systems integrating data-driven AI-based components, rapid responses to deployment issues require fast access to related data for testing and reconfiguration. In the context of automated driving, this especially applies to road obstacles that were not included in the training data, commonly referred to as out-of-distribution (OoD) road obstacles. Given the availability of large uncurated recordings of driving scenes, a pragmatic approach is to query a database to retrieve similar scenarios featuring the same safety concerns due to OoD road obstacles. In this work, we extend beyond identifying OoD road obstacles in video streams and offer a comprehensive approach to extract sequences of OoD road obstacles using text queries, thereby proposing a way of curating a collection of OoD data for subsequent analysis. Our proposed method leverages the recent advances in OoD segmentation and multi-modal foundation models to identify and efficiently extract safety-relevant scenes from unlabeled videos. We present a first approach for the novel task of text-based OoD object retrieval, which addresses the question ''Have we ever encountered this before?''.