Cross apprenticeship learning framework: Properties and solution approaches

TL;DR

This paper introduces the cross apprenticeship learning (CAL) framework, which optimizes policies across multiple environments with different dynamics, balancing environment-specific performance and generalization, supported by theoretical properties and a convex approximation.

Contribution

The paper proposes the CAL framework that balances environment-specific and general policies, providing theoretical insights and a convex approximation for a nonconvex optimization problem.

Findings

Properties of the optimizer vary with the tuning parameter.

The convex outer approximation facilitates solving the nonconvex problem.

Demonstrated effectiveness in a windy gridworld navigation task.

Abstract

Apprenticeship learning is a framework in which an agent learns a policy to perform a given task in an environment using example trajectories provided by an expert. In the real world, one might have access to expert trajectories in different environments where the system dynamics is different while the learning task is the same. For such scenarios, two types of learning objectives can be defined. One where the learned policy performs very well in one specific environment and another when it performs well across all environments. To balance these two objectives in a principled way, our work presents the cross apprenticeship learning (CAL) framework. This consists of an optimization problem where an optimal policy for each environment is sought while ensuring that all policies remain close to each other. This nearness is facilitated by one tuning parameter in the optimization problem. We derive properties of the optimizers of the problem as the tuning parameter varies. Since the problem is nonconvex, we provide a convex outer approximation. Finally, we demonstrate the attributes of our framework in the context of a navigation task in a windy gridworld environment.