Toward a Scientific Discovery Engine for Weather and Climate Data: A Visual Analytics Workbench for Embedding-Based Exploration

TL;DR

This paper introduces an open-source visual analytics workbench that enables scientists to explore, compare, and verify embedding-based representations of weather and climate data, linking latent space results to physical meteorological phenomena.

Contribution

The workbench provides a novel tool for inspecting and comparing embedding models, facilitating scientific discovery in large meteorological datasets by connecting latent space results with physical data.

Findings

Demonstrated tropical-cyclone retrieval using ERA5 embeddings and IBTrACS metadata.

Showed the system's ability to search large embedding collections beyond in-memory limits.

Enabled tracing of latent space results back to source data and physical evidence.

Abstract

Earth system science is producing increasingly large, high-dimensional datasets from physics based Earth system models to AI-based weather and climate models. Embedding-based representations can make these data searchable through similarity search and analog retrieval, but nearest neighbors in latent space are not automatically scientifically meaningful: it may reflect real weather structure, or preprocessing, geography, or model bias. Researchers therefore need ways to inspect how embeddings organize meteorological data, compare representation models, develop retrieval strategies, and verify results against physical evidence. We present an open-source visual analytics workbench for each of these steps. The system links embedding experiments to source data, metadata, spatial context, and model configurations, so latent-space results can be traced back to the physics. Users can explore latent spaces for different models, issue global or localized queries, and inspect analogs through familiar meteorological views. This enables a discovery workflow in which scientists characterize a phenomenon of interest in a well-understood dataset, identifying its signature in latent space, and then use that signature to probe larger, less-labeled archives or ensembles for similar events. We demonstrate the workbench through tropical-cyclone retrieval using ERA5-derived embeddings and IBTrACS metadata, and evaluate its out-of-core retrieval backend to show that large embedding collections can be searched beyond in-memory limits on commodity workstation hardware.