Inspiration Learning through Preferences

TL;DR

This paper introduces Inspiration Learning, a novel imitation learning framework that enables knowledge transfer between agents with different action spaces using preference-based reinforcement learning and a specialized actor-critic architecture.

Contribution

It proposes a new approach to imitation learning that does not require shared action spaces, utilizing a classifier-based reward and an adapted actor-critic method.

Findings

Successfully extends imitation learning to different action spaces

Capable of continuous-to-discrete and primitive-to-macro imitation

Demonstrates effective transfer in diverse agent configurations

Abstract

Current imitation learning techniques are too restrictive because they require the agent and expert to share the same action space. However, oftentimes agents that act differently from the expert can solve the task just as good. For example, a person lifting a box can be imitated by a ceiling mounted robot or a desktop-based robotic-arm. In both cases, the end goal of lifting the box is achieved, perhaps using different strategies. We denote this setup as \textit{Inspiration Learning} - knowledge transfer between agents that operate in different action spaces. Since state-action expert demonstrations can no longer be used, Inspiration learning requires novel methods to guide the agent towards the end goal. In this work, we rely on ideas of Preferential based Reinforcement Learning (PbRL) to design Advantage Actor-Critic algorithms for solving inspiration learning tasks. Unlike classic actor-critic architectures, the critic we use consists of two parts: a) a state-value estimation as in common actor-critic algorithms and b) a single step reward function derived from an expert/agent classifier. We show that our method is capable of extending the current imitation framework to new horizons. This includes continuous-to-discrete action imitation, as well as primitive-to-macro action imitation.