ScenePilot: Controllable Boundary-Driven Critical Scenario Generation for Autonomous Driving

TL;DR

ScenePilot is a novel framework for generating physically feasible yet challenging scenarios for autonomous driving, improving safety testing by focusing on boundary cases that cause failures.

Contribution

It introduces a boundary-driven, reinforcement learning-based approach that balances physical feasibility with the likelihood of causing autonomous vehicle failures.

Findings

ScenePilot increases collision rates by +6.2 percentage points on SafeBench.

It maintains physical validity while generating challenging scenarios.

Adversarial fine-tuning on generated scenarios reduces crash rates.

Abstract

Safety-critical scenarios are central to evaluating autonomous driving systems, yet their rarity in naturalistic logs makes simulation-based stress testing indispensable. Most scenario generation methods treat surrounding agents as adversaries, but they either (i) induce failures without explicitly modeling vehicle-road physical limits, yielding visually extreme yet physically unsolvable crashes, or (ii) enforce physical feasibility or policy feasibility in isolation, which can over-focus on aggressive maneuvers or remain tied to a controller-dependent capability boundary. We propose ScenePilot, a feasibility-guided, boundary-driven framework that targets the boundary band: scenarios that are physically solvable in principle yet still cause the deployed autonomy stack to fail. We formulate generation as constrained multi-objective reinforcement learning, combining an RSS-derived physical-feasibility score $\sigma$ with an online-learned AV-risk predictor $\Phi$, and introduce step-level feasibility-aware shielding to keep exploration near the feasibility boundary while avoiding infeasible artifacts. Experiments on SafeBench with multiple planners show that ScenePilot yields substantially higher collision rates (+6.2 percentage points) while preserving physical validity, and that adversarial fine-tuning on these boundary-band scenarios consistently reduces downstream crash rates. The code is available at https://github.com/QiyuRuan/ScenePilot.