Enhancing Traffic Scene Predictions with Generative Adversarial Networks

TL;DR

This paper introduces a two-stage pipeline combining video prediction and GAN-based enhancement to generate realistic traffic scene frames, improving object detection accuracy for autonomous driving.

Contribution

It presents a novel two-stage approach that enhances predicted traffic scene frames using GANs, leading to better object detection performance in autonomous driving scenarios.

Findings

Enhanced frames improve object detection accuracy by about 9%.

Traditional image metrics do not correlate well with detection performance.

The pipeline effectively generates realistic traffic scene predictions.

Abstract

We present a new two-stage pipeline for predicting frames of traffic scenes where relevant objects can still reliably be detected. Using a recent video prediction network, we first generate a sequence of future frames based on past frames. A second network then enhances these frames in order to make them appear more realistic. This ensures the quality of the predicted frames to be sufficient to enable accurate detection of objects, which is especially important for autonomously driving cars. To verify this two-stage approach, we conducted experiments on the Cityscapes dataset. For enhancing, we trained two image-to-image translation methods based on generative adversarial networks, one for blind motion deblurring and one for image super-resolution. All resulting predictions were quantitatively evaluated using both traditional metrics and a state-of-the-art object detection network showing that the enhanced frames appear qualitatively improved. While the traditional image comparison metrics, i.e., MSE, PSNR, and SSIM, failed to confirm this visual impression, the object detection evaluation resembles it well. The best performing prediction-enhancement pipeline is able to increase the average precision values for detecting cars by about 9% for each prediction step, compared to the non-enhanced predictions.