Uncertainty-Guided Enhancement on Driving Perception System via Foundation Models

TL;DR

This paper introduces an uncertainty-guided method that selectively uses foundation models to improve driving perception accuracy while significantly reducing computational costs by only engaging the foundation model when necessary.

Contribution

It presents a novel uncertainty calibration and thresholding approach that minimizes foundation model queries, improving efficiency without sacrificing accuracy.

Findings

Achieves 10-15% improvement in prediction accuracy.

Reduces foundation model queries by 50%.

Enhances prediction stability with temporal inference.

Abstract

Multimodal foundation models offer promising advancements for enhancing driving perception systems, but their high computational and financial costs pose challenges. We develop a method that leverages foundation models to refine predictions from existing driving perception models -- such as enhancing object classification accuracy -- while minimizing the frequency of using these resource-intensive models. The method quantitatively characterizes uncertainties in the perception model's predictions and engages the foundation model only when these uncertainties exceed a pre-specified threshold. Specifically, it characterizes uncertainty by calibrating the perception model's confidence scores into theoretical lower bounds on the probability of correct predictions using conformal prediction. Then, it sends images to the foundation model and queries for refining the predictions only if the theoretical bound of the perception model's outcome is below the threshold. Additionally, we propose a temporal inference mechanism that enhances prediction accuracy by integrating historical predictions, leading to tighter theoretical bounds. The method demonstrates a 10 to 15 percent improvement in prediction accuracy and reduces the number of queries to the foundation model by 50 percent, based on quantitative evaluations from driving datasets.