Learning Visual Planning Models from Partially Observed Images

TL;DR

This paper introduces Recplan, a novel framework for learning visual planning models from partially observed images by learning latent states and transition models, enabling effective planning despite incomplete data.

Contribution

Recplan is the first framework to learn transition and heuristic models from partially observed raw images for visual planning tasks.

Findings

Recplan outperforms state-of-the-art methods in environments with incomplete observations.

The approach effectively learns latent state representations from raw image traces.

It enables classical planning algorithms to operate directly on visual data.

Abstract

There has been increasing attention on planning model learning in classical planning. Most existing approaches, however, focus on learning planning models from structured data in symbolic representations. It is often difficult to obtain such structured data in real-world scenarios. Although a number of approaches have been developed for learning planning models from fully observed unstructured data (e.g., images), in many scenarios raw observations are often incomplete. In this paper, we provide a novel framework, \aType{Recplan}, for learning a transition model from partially observed raw image traces. More specifically, by considering the preceding and subsequent images in a trace, we learn the latent state representations of raw observations and then build a transition model based on such representations. Additionally, we propose a neural-network-based approach to learn a heuristic model that estimates the distance toward a given goal observation. Based on the learned transition model and heuristic model, we implement a classical planner for images. We exhibit empirically that our approach is more effective than a state-of-the-art approach of learning visual planning models in the environment with incomplete observations.