Frozen Forecasting: A Unified Evaluation

TL;DR

This paper introduces a unified framework for evaluating the forecasting capabilities of frozen vision models across diverse tasks and abstraction levels, using trajectory-based and distributional metrics.

Contribution

It proposes a novel evaluation method that assesses entire trajectories in the model's feature space, enabling consistent comparison across various vision models and tasks.

Findings

Forecasting performance correlates with perceptual quality.

Video synthesis models outperform image-based models in forecasting.

Language supervision does not consistently enhance forecasting abilities.

Abstract

Forecasting future events is a fundamental capability for general-purpose systems that plan or act across different levels of abstraction. Yet, evaluating whether a forecast is "correct" remains challenging due to the inherent uncertainty of the future. We propose a unified evaluation framework for assessing the forecasting capabilities of frozen vision backbones across diverse tasks and abstraction levels. Rather than focusing on single time steps, our framework evaluates entire trajectories and incorporates distributional metrics that better capture the multimodal nature of future outcomes. Given a frozen vision model, we train latent diffusion models to forecast future features directly in its representation space, which are then decoded via lightweight, task-specific readouts. This enables consistent evaluation across a suite of diverse tasks while isolating the forecasting capacity of the backbone itself. We apply our framework to nine diverse vision models, spanning image and video pretraining, contrastive and generative objectives, and with or without language supervision, and evaluate them on four forecasting tasks, from low-level pixel predictions to high-level object motion. We find that forecasting performance strongly correlates with perceptual quality and that the forecasting abilities of video synthesis models are comparable or exceed those pretrained in masking regimes across all levels of abstraction. However, language supervision does not consistently improve forecasting. Notably, video-pretrained models consistently outperform image-based ones.