Learn Proportional Derivative Controllable Latent Space from Pixels

TL;DR

This paper introduces a method to learn a latent space from pixel data that is controllable using simple PD controllers, enabling real-time vision-based control with improved robustness and efficiency.

Contribution

The paper proposes a novel learning objective that ensures the latent space is proportional derivative controllable, facilitating direct application of PD controllers in pixel-based control tasks.

Findings

Outperforms baseline methods in goal reaching tasks

Enables real-time control with simple PD controllers

Achieves robust trajectory tracking in various environments

Abstract

Recent advances in latent space dynamics model from pixels show promising progress in vision-based model predictive control (MPC). However, executing MPC in real time can be challenging due to its intensive computational cost in each timestep. We propose to introduce additional learning objectives to enforce that the learned latent space is proportional derivative controllable. In execution time, the simple PD-controller can be applied directly to the latent space encoded from pixels, to produce simple and effective control to systems with visual observations. We show that our method outperforms baseline methods to produce robust goal reaching and trajectory tracking in various environments.