Enabling Off-Policy Imitation Learning with Deep Actor Critic Stabilization

TL;DR

This paper introduces an off-policy adversarial imitation learning algorithm that enhances sample efficiency and stability by integrating double Q networks and value learning without reward inference.

Contribution

It presents a novel off-policy imitation learning method combining stabilization techniques, reducing sample complexity compared to prior on-policy approaches.

Findings

Reduced sample requirements for expert behavior matching

Enhanced stability through double Q network integration

Effective imitation without reward function inference

Abstract

Learning complex policies with Reinforcement Learning (RL) is often hindered by instability and slow convergence, a problem exacerbated by the difficulty of reward engineering. Imitation Learning (IL) from expert demonstrations bypasses this reliance on rewards. However, state-of-the-art IL methods, exemplified by Generative Adversarial Imitation Learning (GAIL)Ho et. al, suffer from severe sample inefficiency. This is a direct consequence of their foundational on-policy algorithms, such as TRPO Schulman et.al. In this work, we introduce an adversarial imitation learning algorithm that incorporates off-policy learning to improve sample efficiency. By combining an off-policy framework with auxiliary techniques specifically, in this case a double Q network based stabilization and value learning without reward function inference we demonstrate a reduction in the samples required to robustly match expert behavior.