AGCD: Agent-Guided Cross-Modal Decoding for Weather Forecasting

TL;DR

AGCD introduces a novel, controllable, and reusable decoding-time prior-injection method for weather forecasting that enhances accuracy and stability by incorporating state-conditioned physics-priors derived from multivariate atmospheric data.

Contribution

The paper presents AGCD, a plug-and-play framework that injects state-conditioned physics-priors into weather forecasters using multi-agent narration and cross-modal decoding, improving long-term forecast stability.

Findings

Consistent 6-hour forecast improvements across resolutions and backbones.

Reduces early-stage error accumulation in autoregressive rollouts.

Enhances long-horizon stability in weather predictions.

Abstract

Accurate weather forecasting is more than grid-wise regression: it must preserve coherent synoptic structures and physical consistency of meteorological fields, especially under autoregressive rollouts where small one-step errors can amplify into structural bias. Existing physics-priors approaches typically impose global, once-for-all constraints via architectures, regularization, or NWP coupling, offering limited state-adaptive and sample-specific controllability at deployment. To bridge this gap, we propose Agent-Guided Cross-modal Decoding (AGCD), a plug-and-play decoding-time prior-injection paradigm that derives state-conditioned physics-priors from the current multivariate atmosphere and injects them into forecasters in a controllable and reusable way. Specifically, We design a multi-agent meteorological narration pipeline to generate state-conditioned physics-priors, utilizing MLLMs to extract various meteorological elements effectively. To effectively apply the priors, AGCD further introduce cross-modal region interaction decoding that performs region-aware multi-scale tokenization and efficient physics-priors injection to refine visual features without changing the backbone interface. Experiments on WeatherBench demonstrate consistent gains for 6-hour forecasting across two resolutions (5.625 degree and 1.40625 degree) and diverse backbones (generic and weather-specialized), including strictly causal 48-hour autoregressive rollouts that reduce early-stage error accumulation and improve long-horizon stability.