Going Beyond Expert Performance via Deep Implicit Imitation Reinforcement Learning

TL;DR

This paper introduces a deep implicit imitation reinforcement learning framework that enables agents to learn from observation-only datasets, surpass suboptimal expert performance, and handle different action spaces, demonstrated by significant empirical improvements.

Contribution

The paper presents the DIIQN algorithm for imitation from observations and extends it with HA-DIIQN to manage heterogeneous action spaces, addressing key limitations in prior methods.

Findings

DIIQN achieves up to 130% higher episodic returns than standard DQN.

DIIQN outperforms existing implicit imitation methods that can't surpass expert performance.

HA-DIIQN learns up to 64% faster in heterogeneous action scenarios.

Abstract

Imitation learning traditionally requires complete state-action demonstrations from optimal or near-optimal experts. These requirements severely limit practical applicability, as many real-world scenarios provide only state observations without corresponding actions and expert performance is often suboptimal. In this paper we introduce a deep implicit imitation reinforcement learning framework that addresses both limitations by combining deep reinforcement learning with implicit imitation learning from observation-only datasets. Our main algorithm, Deep Implicit Imitation Q-Network (DIIQN), employs an action inference mechanism that reconstructs expert actions through online exploration and integrates a dynamic confidence mechanism that adaptively balances expert-guided and self-directed learning. This enables the agent to leverage expert guidance for accelerated training while maintaining capacity to surpass suboptimal expert performance. We further extend our framework with a Heterogeneous Actions DIIQN (HA-DIIQN) algorithm to tackle scenarios where expert and agent possess different action sets, a challenge previously unaddressed in the implicit imitation learning literature. HA-DIIQN introduces an infeasibility detection mechanism and a bridging procedure identifying alternative pathways connecting agent capabilities to expert guidance when direct action replication is impossible. Our experimental results demonstrate that DIIQN achieves up to 130% higher episodic returns compared to standard DQN, while consistently outperforming existing implicit imitation methods that cannot exceed expert performance. In heterogeneous action settings, HA-DIIQN learns up to 64% faster than baselines, leveraging expert datasets unusable by conventional approaches. Extensive parameter sensitivity analysis reveals the framework's robustness across varying dataset sizes and hyperparameter configurations.