QMDP-Net: Deep Learning for Planning under Partial Observability

TL;DR

QMDP-net is a neural network architecture that integrates planning algorithms for partial observability, enabling end-to-end training and generalization across tasks, with strong simulation performance.

Contribution

It introduces a differentiable neural network that embeds planning algorithms for partial observability, allowing for learning and transfer across multiple tasks.

Findings

Strong performance on robotic simulation tasks

Outperforms traditional QMDP algorithm in experiments

Capable of generalizing to new and similar tasks

Abstract

This paper introduces the QMDP-net, a neural network architecture for planning under partial observability. The QMDP-net combines the strengths of model-free learning and model-based planning. It is a recurrent policy network, but it represents a policy for a parameterized set of tasks by connecting a model with a planning algorithm that solves the model, thus embedding the solution structure of planning in a network learning architecture. The QMDP-net is fully differentiable and allows for end-to-end training. We train a QMDP-net on different tasks so that it can generalize to new ones in the parameterized task set and "transfer" to other similar tasks beyond the set. In preliminary experiments, QMDP-net showed strong performance on several robotic tasks in simulation. Interestingly, while QMDP-net encodes the QMDP algorithm, it sometimes outperforms the QMDP algorithm in the experiments, as a result of end-to-end learning.