LSA: Localized Semantic Alignment for Enhancing Temporal Consistency in Traffic Video Generation

TL;DR

This paper introduces Localized Semantic Alignment (LSA), a fine-tuning framework that improves temporal consistency in traffic video generation by aligning semantic features, eliminating the need for control signals during inference.

Contribution

We propose LSA, a novel fine-tuning method that enhances temporal consistency in pre-trained video generation models through semantic feature alignment around dynamic objects.

Findings

LSA outperforms baselines in standard video generation metrics.

The approach improves temporal consistency without additional inference overhead.

Effective on nuScenes and KITTI datasets.

Abstract

Controllable video generation has emerged as a versatile tool for autonomous driving, enabling realistic synthesis of traffic scenarios. However, existing methods depend on control signals at inference time to guide the generative model towards temporally consistent generation of dynamic objects, limiting their utility as scalable and generalizable data engines. In this work, we propose Localized Semantic Alignment (LSA), a simple yet effective framework for fine-tuning pre-trained video generation models. LSA enhances temporal consistency by aligning semantic features between ground-truth and generated video clips. Specifically, we compare the output of an off-the-shelf feature extraction model between the ground-truth and generated video clips localized around dynamic objects inducing a semantic feature consistency loss. We fine-tune the base model by combining this loss with the standard diffusion loss. The model fine-tuned for a single epoch with our novel loss outperforms the baselines in common video generation evaluation metrics. To further test the temporal consistency in generated videos we adapt two additional metrics from object detection task, namely mAP and mIoU. Extensive experiments on nuScenes and KITTI datasets show the effectiveness of our approach in enhancing temporal consistency in video generation without the need for external control signals during inference and any computational overheads.