Optimization-Guided Diffusion for Interactive Scene Generation

TL;DR

OMEGA is an optimization-guided diffusion framework that enhances the realism, safety, and controllability of synthetic multi-agent driving scenes for autonomous vehicle testing.

Contribution

It introduces a training-free, constrained optimization approach to enforce physical and social constraints during diffusion-based scene generation, including game-theoretic modeling of adversarial interactions.

Findings

Increases scene validity from 32.35% to 72.27%.

Raises controllability-focused scene validity from 11% to 80%.

Generates 5 times more near-collision frames with quick time-to-collision.

Abstract

Realistic and diverse multi-agent driving scenes are crucial for evaluating autonomous vehicles, but safety-critical events which are essential for this task are rare and underrepresented in driving datasets. Data-driven scene generation offers a low-cost alternative by synthesizing complex traffic behaviors from existing driving logs. However, existing models often lack controllability or yield samples that violate physical or social constraints, limiting their usability. We present OMEGA, an optimization-guided, training-free framework that enforces structural consistency and interaction awareness during diffusion-based sampling from a scene generation model. OMEGA re-anchors each reverse diffusion step via constrained optimization, steering the generation towards physically plausible and behaviorally coherent trajectories. Building on this framework, we formulate ego-attacker interactions as a game-theoretic optimization in the distribution space, approximating Nash equilibria to generate realistic, safety-critical adversarial scenarios. Experiments on nuPlan and Waymo show that OMEGA improves generation realism, consistency, and controllability, increasing the ratio of physically and behaviorally valid scenes from 32.35% to 72.27% for free exploration capabilities, and from 11% to 80% for controllability-focused generation. Our approach can also generate $5\times$ more near-collision frames with a time-to-collision under three seconds while maintaining the overall scene realism.