Generalized Inverse Planning: Learning Lifted non-Markovian Utility for Generalizable Task Representation

TL;DR

This paper introduces a new framework called Generalized Inverse Planning that learns non-Markovian, lifted utility functions from demonstrations, enabling generalizable task representations across different environments and structural changes.

Contribution

It proposes a novel generalization framework for utility learning that extends beyond traditional IRL, incorporating non-Markovian and lifted utilities for better transferability.

Findings

MEIP effectively learns non-Markovian utility functions from demonstrations.

The approach generalizes across different planning problems and structural changes.

Successful application to folding tasks demonstrates practical utility.

Abstract

In searching for a generalizable representation of temporally extended tasks, we spot two necessary constituents: the utility needs to be non-Markovian to transfer temporal relations invariant to a probability shift, the utility also needs to be lifted to abstract out specific grounding objects. In this work, we study learning such utility from human demonstrations. While inverse reinforcement learning (IRL) has been accepted as a general framework of utility learning, its fundamental formulation is one concrete Markov Decision Process. Thus the learned reward function does not specify the task independently of the environment. Going beyond that, we define a domain of generalization that spans a set of planning problems following a schema. We hence propose a new quest, Generalized Inverse Planning, for utility learning in this domain. We further outline a computational framework, Maximum Entropy Inverse Planning (MEIP), that learns non-Markovian utility and associated concepts in a generative manner. The learned utility and concepts form a task representation that generalizes regardless of probability shift or structural change. Seeing that the proposed generalization problem has not been widely studied yet, we carefully define an evaluation protocol, with which we illustrate the effectiveness of MEIP on two proof-of-concept domains and one challenging task: learning to fold from demonstrations.