Case-Based Inverse Reinforcement Learning Using Temporal Coherence

TL;DR

This paper introduces a case-based inverse reinforcement learning algorithm that leverages temporal coherence to learn high-level strategies from minimal expert data, improving stability and efficiency in imitation learning.

Contribution

The novel approach focuses on imitating the expert's higher-level strategy rather than actions, using temporal coherence to train a neural network for state similarity prediction.

Findings

Learns near-optimal policies with very little expert data

Outperforms action-level imitation algorithms in data-scarce settings

Uses temporal coherence to improve imitation stability

Abstract

Providing expert trajectories in the context of Imitation Learning is often expensive and time-consuming. The goal must therefore be to create algorithms which require as little expert data as possible. In this paper we present an algorithm that imitates the higher-level strategy of the expert rather than just imitating the expert on action level, which we hypothesize requires less expert data and makes training more stable. As a prior, we assume that the higher-level strategy is to reach an unknown target state area, which we hypothesize is a valid prior for many domains in Reinforcement Learning. The target state area is unknown, but since the expert has demonstrated how to reach it, the agent tries to reach states similar to the expert. Building on the idea of Temporal Coherence, our algorithm trains a neural network to predict whether two states are similar, in the sense that they may occur close in time. During inference, the agent compares its current state with expert states from a Case Base for similarity. The results show that our approach can still learn a near-optimal policy in settings with very little expert data, where algorithms that try to imitate the expert at the action level can no longer do so.