LAOF: Robust Latent Action Learning with Optical Flow Constraints

TL;DR

LAOF introduces a pseudo-supervised framework utilizing optical flow to learn robust latent action representations from videos, effectively handling distractors and reducing reliance on labeled data, thus enhancing downstream imitation and reinforcement learning.

Contribution

The paper presents LAOF, a novel optical flow-based pseudo-supervised method for latent action learning that is robust to distractors and effective with minimal or no action labels.

Findings

LAOF outperforms existing methods on downstream tasks.

Optical flow constraints stabilize training and improve representation quality.

LAOF matches or surpasses fully supervised methods with only 1% action labels.

Abstract

Learning latent actions from large-scale videos is crucial for the pre-training of scalable embodied foundation models, yet existing methods often struggle with action-irrelevant distractors. Although incorporating action supervision can alleviate these distractions, its effectiveness is restricted by the scarcity of available action labels. Optical flow represents pixel-level motion between consecutive frames, naturally suppressing background elements and emphasizing moving objects. Motivated by this, we propose robust Latent Action learning with Optical Flow constraints, called LAOF, a pseudo-supervised framework that leverages the agent's optical flow as an action-driven signal to learn latent action representations robust to distractors. Experimental results show that the latent representations learned by LAOF outperform existing methods on downstream imitation learning and reinforcement learning tasks. This superior performance arises from optical flow constraints, which substantially stabilize training and improve the quality of latent representations under extremely label-scarce conditions, while remaining effective as the proportion of action labels increases to 10 percent. Importantly, even without action supervision, LAOF matches or surpasses action-supervised methods trained with 1 percent of action labels.