Using the SAL technique for spatial verification of cloud processes: A sensitivity analysis

TL;DR

This study evaluates the SAL spatial verification method applied to cloud data, analyzing its sensitivity to object identification parameters and observational uncertainties, and proposing indicators to assess score sensitivity.

Contribution

It demonstrates the high sensitivity of SAL scores to threshold parameters and introduces new indicators to evaluate this sensitivity without extensive computations.

Findings

SAL scores are highly sensitive to threshold levels.

Small parameter changes can cause large score variations.

Proposed sensitivity indicators effectively assess score robustness.

Abstract

The feature based spatial verification method SAL is applied to cloud data, i.e. two-dimensional spatial fields of total cloud cover and spectral radiance. Model output is obtained from the COSMO-DE forward operator SynSat and compared to SEVIRI satellite data. The aim of this study is twofold. First, to assess the applicability of SAL to this kind of data, and second, to analyze the role of external object identification algorithms (OIA) and the effects of observational uncertainties on the resulting scores. As a feature based method, SAL requires external OIA. A comparison of three different algorithms shows that the threshold level, which is a fundamental part of all studied algorithms, induces high sensitivity and unstable behavior of object dependent SAL scores (i.e. even very small changes in parameter values can lead to large changes in the resulting scores). An in-depth statistical analysis reveals significant effects on distributional quantities commonly used in the interpretation of SAL, e.g. median and interquartile distance. Two sensitivity indicators based on the univariate cumulative distribution functions are derived. They allow to asses the sensitivity of the SAL scores to threshold level changes without computationally expensive iterative calculations of SAL for various thresholds. The mathematical structure of these indicators connects the sensitivity of the SAL scores to parameter changes with the effect of observational uncertainties. Finally, the discriminating power of SAL is studied. It is shown, that - for large-scale cloud data - changes in the parameters may have larger effects on the object dependent SAL scores (i.e. the S and L2 scores) than a complete loss of temporal collocation.