Personalized Adapter for Large Meteorology Model on Devices: Towards Weather Foundation Models

TL;DR

This paper introduces LM-Weather, a personalized adapter approach that adapts pre-trained language models for efficient, on-device meteorological data modeling, achieving high accuracy and customization while preserving privacy.

Contribution

The paper proposes a lightweight personalized adapter for PLMs, enabling effective on-device meteorological modeling with high efficiency and privacy, outperforming state-of-the-art methods.

Findings

LM-Weather outperforms existing methods in forecasting and imputation tasks.

The approach enables highly customized models for heterogeneous devices.

It generalizes well in data-limited and out-of-distribution scenarios.

Abstract

This paper demonstrates that pre-trained language models (PLMs) are strong foundation models for on-device meteorological variables modeling. We present LM-Weather, a generic approach to taming PLMs, that have learned massive sequential knowledge from the universe of natural language databases, to acquire an immediate capability to obtain highly customized models for heterogeneous meteorological data on devices while keeping high efficiency. Concretely, we introduce a lightweight personalized adapter into PLMs and endows it with weather pattern awareness. During communication between clients and the server, low-rank-based transmission is performed to effectively fuse the global knowledge among devices while maintaining high communication efficiency and ensuring privacy. Experiments on real-wold dataset show that LM-Weather outperforms the state-of-the-art results by a large margin across various tasks (e.g., forecasting and imputation at different scales). We provide extensive and in-depth analyses experiments, which verify that LM-Weather can (1) indeed leverage sequential knowledge from natural language to accurately handle meteorological sequence, (2) allows each devices obtain highly customized models under significant heterogeneity, and (3) generalize under data-limited and out-of-distribution (OOD) scenarios.