Imitation Learning from a Single Temporally Misaligned Video

TL;DR

This paper introduces ORCA, a sequence-level reward function for imitation learning from a single, temporally misaligned demonstration, significantly improving task performance over frame-level methods.

Contribution

The paper proposes ORCA, a novel sequence-based alignment method that enforces correct temporal ordering, addressing limitations of frame-level matching in imitation learning from misaligned demonstrations.

Findings

ORCA achieves 4.5x improvement on Meta-world tasks.

ORCA achieves 6.6x improvement on Humanoid-v4 tasks.

Empirical analysis confirms ORCA's robustness to temporal misalignment.

Abstract

We examine the problem of learning sequential tasks from a single visual demonstration. A key challenge arises when demonstrations are temporally misaligned due to variations in timing, differences in embodiment, or inconsistencies in execution. Existing approaches treat imitation as a distribution-matching problem, aligning individual frames between the agent and the demonstration. However, we show that such frame-level matching fails to enforce temporal ordering or ensure consistent progress. Our key insight is that matching should instead be defined at the level of sequences. We propose that perfect matching occurs when one sequence successfully covers all the subgoals in the same order as the other sequence. We present ORCA (ORdered Coverage Alignment), a dense per-timestep reward function that measures the probability of the agent covering demonstration frames in the correct order. On temporally misaligned demonstrations, we show that agents trained with the ORCA reward achieve $4.5$x improvement ($0.11 \rightarrow 0.50$ average normalized returns) for Meta-world tasks and $6.6$x improvement ($6.55 \rightarrow 43.3$ average returns) for Humanoid-v4 tasks compared to the best frame-level matching algorithms. We also provide empirical analysis showing that ORCA is robust to varying levels of temporal misalignment. Our code is available at https://github.com/portal-cornell/orca/