Privileged Foresight Distillation: Zero-Cost Future Correction for World Action Models

TL;DR

This paper introduces Privileged Foresight Distillation (PFD), a method that distills future-conditioned corrections into current-only models, improving manipulation benchmarks without increasing inference complexity.

Contribution

The paper proposes a novel residual-based distillation approach that leverages privileged future observations to enhance current-only policies.

Findings

PFD improves performance on LIBERO and RoboTwin benchmarks.

Future video is never generated at inference, maintaining low latency.

The approach captures genuine future-conditioned corrections, not just regularization effects.

Abstract

World action models jointly predict future video and action during training, raising an open question about what role the future-prediction branch actually plays. A recent finding shows that this branch can be removed at inference with little to no loss on common manipulation benchmarks, suggesting that future information may act merely as a regularizer on the shared visual backbone. We propose instead that joint training induces an action-conditioned correction that privileged future observations impose on action denoising, and that current-only policies capture this correction only partially. Making the account precise, we formulate privileged foresight as a residual in the action-denoising direction -- the difference between what a model predicts given the true future and what it predicts given only the current frame -- and introduce \emph{Privileged Foresight Distillation (PFD)}, which transfers this residual from a training-time teacher into a small adapter on a current-only student. The teacher and student share the same backbone and differ only in the attention mask over video tokens; future video is never generated at inference. Controlled experiments verify that this gain reflects a genuine future-conditioned correction rather than a side effect of capacity or regularization. Empirically, PFD achieves consistent improvements on LIBERO and RoboTwin manipulation benchmarks while preserving the current-only inference interface at negligible added latency. This view reframes the role of future information in world action models: not as a target to predict, nor as a regularizer to absorb, but as a compressible correction to be distilled.