ClimateIQA: A New Dataset and Benchmark to Advance Vision-Language Models in Meteorology Anomalies Analysis

TL;DR

ClimateIQA introduces a new dataset and benchmark for vision-language models to improve analysis of meteorological heatmaps, addressing challenges in irregular shapes and complex patterns with a novel algorithm and specialized training.

Contribution

The paper presents SPOT, a novel algorithm for processing irregular visual regions, and ClimateIQA, a large meteorological VQA dataset, advancing model understanding of weather heatmaps.

Findings

ClimateZOO models outperform existing models in heatmap tasks.

SPOT effectively localizes irregularly shaped regions in visual data.

ClimateIQA enhances VLM accuracy in meteorological analysis.

Abstract

Meteorological heatmaps play a vital role in deciphering extreme weather phenomena, yet their inherent complexities marked by irregular contours, unstructured patterns, and complex color variations present unique analytical hurdles for state-of-the-art Vision-Language Models (VLMs). Current state-of-the-art models like GPT-4o, Qwen-VL, and LLaVA 1.6 struggle with tasks such as precise color identification and spatial localization, resulting in inaccurate or incomplete interpretations. To address these challenges, we introduce Sparse Position and Outline Tracking (SPOT), a novel algorithm specifically designed to process irregularly shaped colored regions in visual data. SPOT identifies and localizes these regions by extracting their spatial coordinates, enabling structured representations of irregular shapes. Building on SPOT, we construct ClimateIQA, a novel meteorological visual question answering (VQA) dataset, comprising 26,280 high-resolution heatmaps and 762,120 instruction samples for wind gust, total precipitation, wind chill index and heat index analysis. ClimateIQA enhances VLM training by incorporating spatial cues, geographic metadata, and reanalysis data, improving model accuracy in interpreting and describing extreme weather features. Furthermore, we develop Climate-Zoo, a suite of fine-tuned VLMs based on SPOT-empowered ClimateIQA, which significantly outperforms existing models in meteorological heatmap tasks.