Practical Issues of Action-conditioned Next Image Prediction

TL;DR

This paper compares models for action-conditioned image prediction in autonomous driving, highlighting the importance of action tiling encoding and proposing a lightweight, efficient model with competitive performance.

Contribution

It demonstrates the critical role of action tiling encoding in model performance and introduces a smaller, efficient model suitable for real-time autonomous vehicle applications.

Findings

Action tiling encoding significantly improves prediction accuracy.

The lightweight model achieves comparable results to larger models.

The proposed model is more memory-efficient and computationally suitable for self-driving cars.

Abstract

The problem of action-conditioned image prediction is to predict the expected next frame given the current camera frame the robot observes and an action selected by the robot. We provide the first comparison of two recent popular models, especially for image prediction on cars. Our major finding is that action tiling encoding is the most important factor leading to the remarkable performance of the CDNA model. We present a light-weight model by action tiling encoding which has a single-decoder feedforward architecture same as [action_video_prediction_honglak]. On a real driving dataset, the CDNA model achieves ${0.3986} \times 10^{-3}$ MSE and ${0.9846}$ Structure SIMilarity (SSIM) with a network size of about {\bfseries ${12.6}$ million} parameters. With a small network of fewer than {\bfseries ${1}$ million} parameters, our new model achieves a comparable performance to CDNA at ${0.3613} \times 10^{-3}$ MSE and ${0.9633}$ SSIM. Our model requires less memory, is more computationally efficient and is advantageous to be used inside self-driving vehicles.