Visual Imitation Learning with Patch Rewards

TL;DR

This paper introduces PatchAIL, a novel visual imitation learning method that uses patch-based rewards to improve learning accuracy and interpretability from visual demonstrations, outperforming existing methods.

Contribution

It proposes a patch-based discriminator to measure local expertise in images, enabling fine-grained rewards and enhanced training stability in imitation learning.

Findings

PatchAIL outperforms baseline methods on DeepMind Control and Atari tasks.

Patch rewards provide better interpretability of visual demonstrations.

PatchAIL improves training stability through regularization.

Abstract

Visual imitation learning enables reinforcement learning agents to learn to behave from expert visual demonstrations such as videos or image sequences, without explicit, well-defined rewards. Previous research either adopted supervised learning techniques or induce simple and coarse scalar rewards from pixels, neglecting the dense information contained in the image demonstrations. In this work, we propose to measure the expertise of various local regions of image samples, or called \textit{patches}, and recover multi-dimensional \textit{patch rewards} accordingly. Patch reward is a more precise rewarding characterization that serves as a fine-grained expertise measurement and visual explainability tool. Specifically, we present Adversarial Imitation Learning with Patch Rewards (PatchAIL), which employs a patch-based discriminator to measure the expertise of different local parts from given images and provide patch rewards. The patch-based knowledge is also used to regularize the aggregated reward and stabilize the training. We evaluate our method on DeepMind Control Suite and Atari tasks. The experiment results have demonstrated that PatchAIL outperforms baseline methods and provides valuable interpretations for visual demonstrations.