From Failures to Fixes: LLM-Driven Scenario Repair for Self-Evolving Autonomous Driving

TL;DR

This paper introduces SERA, an LLM-powered framework that enables autonomous driving systems to self-evolve by repairing failure cases through targeted scenario recommendation and fine-tuning, improving safety-critical performance.

Contribution

The paper presents a novel LLM-driven approach for self-evolving autonomous driving systems that intelligently repairs failure cases with minimal data, enhancing robustness and safety.

Findings

SERA improves key autonomous driving metrics across multiple baselines.

The framework effectively identifies failure patterns and retrieves relevant scenarios.

Targeted fine-tuning with SERA enhances safety-critical performance.

Abstract

Ensuring robust and generalizable autonomous driving requires not only broad scenario coverage but also efficient repair of failure cases, particularly those related to challenging and safety-critical scenarios. However, existing scenario generation and selection methods often lack adaptivity and semantic relevance, limiting their impact on performance improvement. In this paper, we propose \textbf{SERA}, an LLM-powered framework that enables autonomous driving systems to self-evolve by repairing failure cases through targeted scenario recommendation. By analyzing performance logs, SERA identifies failure patterns and dynamically retrieves semantically aligned scenarios from a structured bank. An LLM-based reflection mechanism further refines these recommendations to maximize relevance and diversity. The selected scenarios are used for few-shot fine-tuning, enabling targeted adaptation with minimal data. Experiments on the benchmark show that SERA consistently improves key metrics across multiple autonomous driving baselines, demonstrating its effectiveness and generalizability under safety-critical conditions.