RadarQA: Multi-modal Quality Analysis of Weather Radar Forecasts

TL;DR

RadarQA leverages multi-modal large language models to provide detailed, interpretable quality assessments of weather radar forecasts, surpassing traditional metrics and aiding meteorological analysis.

Contribution

Introduces RadarQA, a novel MLLM-based framework for multi-modal weather forecast quality analysis, including a large annotated dataset and a multi-stage training strategy.

Findings

RadarQA outperforms existing models in quality assessment tasks.

The RQA-70K dataset enables robust benchmarking.

Multi-stage training improves model accuracy.

Abstract

Quality analysis of weather forecasts is an essential topic in meteorology. Although traditional score-based evaluation metrics can quantify certain forecast errors, they are still far from meteorological experts in terms of descriptive capability, interpretability, and understanding of dynamic evolution. With the rapid development of Multi-modal Large Language Models (MLLMs), these models become potential tools to overcome the above challenges. In this work, we introduce an MLLM-based weather forecast analysis method, RadarQA, integrating key physical attributes with detailed assessment reports. We introduce a novel and comprehensive task paradigm for multi-modal quality analysis, encompassing both single frame and sequence, under both rating and assessment scenarios. To support training and benchmarking, we design a hybrid annotation pipeline that combines human expert labeling with automated heuristics. With such an annotation method, we construct RQA-70K, a large-scale dataset with varying difficulty levels for radar forecast quality evaluation. We further design a multi-stage training strategy that iteratively improves model performance at each stage. Extensive experiments show that RadarQA outperforms existing general MLLMs across all evaluation settings, highlighting its potential for advancing quality analysis in weather prediction.