Is Optimal Transport Necessary for Inverse Reinforcement Learning?

TL;DR

This paper questions the necessity of optimal transport in inverse reinforcement learning, proposing simple, heuristic alternatives that match or outperform OT-based methods across various benchmarks, emphasizing simplicity and efficiency.

Contribution

The authors introduce two heuristic IRL methods that bypass optimal transport, demonstrating their effectiveness and efficiency through extensive empirical evaluation.

Findings

Heuristic methods match or outperform OT-based approaches.

Simple proximity-based rewards are effective for IRL.

Complex OT optimization may be unnecessary for reward inference.

Abstract

Inverse Reinforcement Learning (IRL) aims to recover a reward function from expert demonstrations. Recently, Optimal Transport (OT) methods have been successfully deployed to align trajectories and infer rewards. While OT-based methods have shown strong empirical results, they introduce algorithmic complexity, hyperparameter sensitivity, and require solving the OT optimization problems. In this work, we challenge the necessity of OT in IRL by proposing two simple, heuristic alternatives: (1) Minimum-Distance Reward, which assigns rewards based on the nearest expert state regardless of temporal order; and (2) Segment-Matching Reward, which incorporates lightweight temporal alignment by matching agent states to corresponding segments in the expert trajectory. These methods avoid optimization, exhibit linear-time complexity, and are easy to implement. Through extensive evaluations across 32 online and offline benchmarks with three reinforcement learning algorithms, we show that our simple rewards match or outperform recent OT-based approaches. Our findings suggest that the core benefits of OT may arise from basic proximity alignment rather than its optimal coupling formulation, advocating for reevaluation of complexity in future IRL design.